Only use the relative path in HTTP requests

This causes a pretty big problem on out-of-spec HTTP servers (like
Flickr). Closes #31

README.md

@@ -52,6 +52,8 @@ import vcr
 my_vcr = vcr.VCR(
     serializer = 'json',
     cassette_library_dir = 'fixtures/cassettes',
+    record_mode = 'once',
+    match_on = ['url', 'method'],
 )
 
 with my_vcr.use_cassette('test.json'):
@@ -61,12 +63,75 @@ with my_vcr.use_cassette('test.json'):
 Otherwise, you can override options each time you use a cassette.  
 
 ```python
-with vcr.use_cassette('test.yml', serializer='json'):
+with vcr.use_cassette('test.yml', serializer='json', record_mode='once'):
     # your http code here
 ```
 
 Note: Per-cassette overrides take precedence over the global config.
 
+## Request matching
+
+Request matching is configurable and allows you to change which requests
+VCR considers identical.  The default behavior is `['url', method']`
+which means that requests with both the same URL and method (ie POST or
+GET) are considered identical.
+
+This can be configured by changing the `match_on` setting.
+
+The following options are available :
+
+ * method (for example, POST or GET)
+ * url (the full URL, including the protocol)
+ * host (the hostname of the server receiving the request)
+ * path (excluding the hostname)
+ * body (the entire request body)
+ * headers (the headers of the request)
+
+If these options don't work for you, you can also register your own
+request matcher.  This is described in the Advanced section of this
+README.
+
+## Record Modes
+VCR supports 4 record modes (with the same behavior as Ruby's VCR):
+
+### once
+
+ * Replay previously recorded interactions.
+ * Record new interactions if there is no cassette file.
+ * Cause an error to be raised for new requests if there is a cassette file.
+ 
+It is similar to the :new_episodes record mode, but will prevent new,
+unexpected requests from being made (i.e. because the request URI
+changed).
+
+once is the default record mode, used when you do not set one.
+
+### new_episodes
+
+* Record new interactions.
+* Replay previously recorded interactions.
+It is similar to the once record mode, but will always record new
+interactions, even if you have an existing recorded one that is similar,
+but not identical.
+
+This was the default behavior in versions < 0.3.0
+
+### none
+
+* Replay previously recorded interactions.
+* Cause an error to be raised for any new requests.
+This is useful when your code makes potentially dangerous
+HTTP requests. The none record mode guarantees that no
+new HTTP requests will be made.
+
+### all
+
+* Record new interactions.
+* Never replay previously recorded interactions.
+This can be temporarily used to force VCR to re-record
+a cassette (i.e. to ensure the responses are not out of date)
+or can be used when you simply want to log all HTTP requests.
+
 ## Advanced Features
 
 If you want, VCR.py can return information about the cassette it is
@@ -122,7 +187,7 @@ you would like.  Create your own module or class instance with 2 methods:
 Finally, register your class with VCR to use your
 new serializer.
 
-```
+```python
 import vcr
 
 BogoSerializer(object):
@@ -146,6 +211,41 @@ with my_vcr.use_cassette('test.bogo'):
 
 ```
 
+## Register your own request matcher
+
+Create your own method with the following signature
+
+```python
+def my_matcher(r1, r2):
+```    
+
+Your method receives the two requests and must return True if they
+match, False if they don't.
+
+Finally, register your method with VCR to use your
+new request matcher.
+
+```python
+import vcr
+
+def jurassic_matcher(r1, r2):
+    return r1.url == r2.url and 'JURASSIC PARK' in r1.body
+
+my_vcr = vcr.VCR()
+my_vcr.register_matcher('jurassic', jurassic_matcher)
+
+with my_vcr.use_cassette('test.yml', match_on=['jurassic']):
+    # your http here
+
+# After you register, you can set the default match_on to use your new matcher
+
+my_vcr.match_on = ['jurassic']
+
+with my_vcr.use_cassette('test.yml'):
+    # your http here
+
+```
+
 ##Installation
 
 VCR.py is a package on PyPI, so you can `pip install vcrpy` (first you may need to `brew install libyaml` [[Homebrew](http://mxcl.github.com/homebrew/)])
@@ -159,6 +259,21 @@ This library is a work in progress, so the API might change on you.
 There are probably some [bugs](https://github.com/kevin1024/vcrpy/issues?labels=bug&page=1&state=open) floating around too.
 
 ##Changelog
+* 0.3.1: Fix issue causing full paths to be sent on the HTTP request
+  line.
+* 0.3.0: *Backwards incompatible release* - Added support for record
+  modes, and changed the default recording behavior to the "once" record
+  mode.  Please see the documentation on record modes for more.  Added
+  support for custom request matching, and changed the default request
+  matching behavior to match only on the URL and method. Also,
+  improved the httplib mocking to add support for the `HTTPConnection.send()`
+  method.  This means that requests won't actually be sent until the
+  response is read, since I need to record the entire request in order
+  to match up the appropriate response.  I don't think this should cause
+  any issues unless you are sending requests without ever loading the
+  response (which none of the standard httplib wrappers do, as far as I
+  know.  Thanks to @fatuhoku for some of the ideas and the motivation 
+  behind this release.
 * 0.2.1: Fixed missing modules in setup.py
 * 0.2.0: Added configuration API, which lets you configure some settings
   on VCR (see the README). Also, VCR no longer saves cassettes if they

setup.py

@@ -19,7 +19,7 @@ class PyTest(TestCommand):
         sys.exit(errno)
 
 setup(name='vcrpy',
-      version='0.2.1',
+      version='0.3.1',
       description="A Python port of Ruby's VCR to make mocking HTTP easier",
       author='Kevin McCarthy',
       author_email='me@kevinmccarthy.org',
@@ -39,10 +39,10 @@ setup(name='vcrpy',
       },
       install_requires=['PyYAML'],
       license='MIT',
-      tests_require=['pytest'],
+      tests_require=['pytest','mock'],
       cmdclass={'test': PyTest},
       classifiers=[
-          'Development Status :: 3 - Alpha',
+          'Development Status :: 4 - Beta',
           'Environment :: Console',
           'Intended Audience :: Developers',
           'Programming Language :: Python',

tests/integration/test_disksaver.py

@@ -45,7 +45,7 @@ def test_disk_saver_write(tmpdir):
     # the mtime doesn't change
     time.sleep(1)
 
-    with vcr.use_cassette(fname) as cass:
+    with vcr.use_cassette(fname, record_mode='any') as cass:
         urllib2.urlopen('http://www.iana.org/domains/reserved').read()
         urllib2.urlopen('http://httpbin.org/').read()
         assert cass.play_count == 1

tests/integration/test_record_mode.py

@@ -0,0 +1,86 @@
+import os
+import urllib2
+import pytest
+import vcr
+
+
+def test_once_record_mode(tmpdir):
+    testfile = str(tmpdir.join('recordmode.yml'))
+    with vcr.use_cassette(testfile, record_mode="once"):
+        # cassette file doesn't exist, so create.
+        response = urllib2.urlopen('http://httpbin.org/').read()
+
+    with vcr.use_cassette(testfile, record_mode="once") as cass:
+        # make the same request again
+        response = urllib2.urlopen('http://httpbin.org/').read()
+
+        # the first time, it's played from the cassette.
+        # but, try to access something else from the same cassette, and an
+        # exception is raised.
+        with pytest.raises(Exception):
+            response = urllib2.urlopen('http://httpbin.org/get').read()
+
+
+def test_new_episodes_record_mode(tmpdir):
+    testfile = str(tmpdir.join('recordmode.yml'))
+
+    with vcr.use_cassette(testfile, record_mode="new_episodes"):
+        # cassette file doesn't exist, so create.
+        response = urllib2.urlopen('http://httpbin.org/').read()
+
+    with vcr.use_cassette(testfile, record_mode="new_episodes") as cass:
+        # make the same request again
+        response = urllib2.urlopen('http://httpbin.org/').read()
+
+        # in the "new_episodes" record mode, we can add more requests to
+        # a cassette without repurcussions.
+        response = urllib2.urlopen('http://httpbin.org/get').read()
+
+        # the first interaction was not re-recorded, but the second was added
+        assert cass.play_count == 1
+
+
+def test_all_record_mode(tmpdir):
+    testfile = str(tmpdir.join('recordmode.yml'))
+
+    with vcr.use_cassette(testfile, record_mode="all"):
+        # cassette file doesn't exist, so create.
+        response = urllib2.urlopen('http://httpbin.org/').read()
+
+    with vcr.use_cassette(testfile, record_mode="all") as cass:
+        # make the same request again
+        response = urllib2.urlopen('http://httpbin.org/').read()
+
+        # in the "all" record mode, we can add more requests to
+        # a cassette without repurcussions.
+        response = urllib2.urlopen('http://httpbin.org/get').read()
+
+        # The cassette was never actually played, even though it existed.
+        # that's because, in "all" mode, the requests all go directly to
+        # the source and bypass the cassette.
+        assert cass.play_count == 0
+
+
+def test_none_record_mode(tmpdir):
+    # Cassette file doesn't exist, yet we are trying to make a request.
+    # raise hell.
+    testfile = str(tmpdir.join('recordmode.yml'))
+    with vcr.use_cassette(testfile, record_mode="none"):
+        with pytest.raises(Exception):
+            response = urllib2.urlopen('http://httpbin.org/').read()
+
+
+def test_none_record_mode_with_existing_cassette(tmpdir):
+    # create a cassette file
+    testfile = str(tmpdir.join('recordmode.yml'))
+
+    with vcr.use_cassette(testfile, record_mode="all"):
+        response = urllib2.urlopen('http://httpbin.org/').read()
+
+    # play from cassette file
+    with vcr.use_cassette(testfile, record_mode="none") as cass:
+        response = urllib2.urlopen('http://httpbin.org/').read()
+        assert cass.play_count == 1
+        # but if I try to hit the net, raise an exception.
+        with pytest.raises(Exception):
+            response = urllib2.urlopen('http://httpbin.org/get').read()

tests/integration/test_register_matcher.py

@@ -0,0 +1,32 @@
+import urllib2
+import vcr
+
+
+def true_matcher(r1, r2):
+    return True
+
+
+def false_matcher(r1, r2):
+    return False
+
+
+def test_registered_serializer_true_matcher(tmpdir):
+    my_vcr = vcr.VCR()
+    my_vcr.register_matcher('true', true_matcher)
+    testfile = str(tmpdir.join('test.yml'))
+    with my_vcr.use_cassette(testfile, match_on=['true']) as cass:
+        # These 2 different urls are stored as the same request
+        urllib2.urlopen('http://httpbin.org/')
+        urllib2.urlopen('https://httpbin.org/get')
+        assert len(cass) == 1
+
+
+def test_registered_serializer_false_matcher(tmpdir):
+    my_vcr = vcr.VCR()
+    my_vcr.register_matcher('false', false_matcher)
+    testfile = str(tmpdir.join('test.yml'))
+    with my_vcr.use_cassette(testfile, match_on=['false']) as cass:
+        # These 2 different urls are stored as different requests
+        urllib2.urlopen('http://httpbin.org/')
+        urllib2.urlopen('https://httpbin.org/get')
+        assert len(cass) == 2

tests/integration/test_requests.py

@@ -84,7 +84,7 @@ def test_post(tmpdir, scheme):
     '''Ensure that we can post and cache the results'''
     data = {'key1': 'value1', 'key2': 'value2'}
     url = scheme + '://httpbin.org/post'
-    with vcr.use_cassette(str(tmpdir.join('redirect.yaml'))) as cass:
+    with vcr.use_cassette(str(tmpdir.join('requests.yaml'))) as cass:
         # Ensure that this is empty to begin with
         assert_cassette_empty(cass)
         req1 = requests.post(url, data).content
@@ -97,7 +97,7 @@ def test_post(tmpdir, scheme):
 def test_redirects(tmpdir, scheme):
     '''Ensure that we can handle redirects'''
     url = scheme + '://httpbin.org/redirect-to?url=bytes/1024'
-    with vcr.use_cassette(str(tmpdir.join('redirect.yaml'))) as cass:
+    with vcr.use_cassette(str(tmpdir.join('requests.yaml'))) as cass:
         # Ensure that this is empty to begin with
         assert_cassette_empty(cass)
         assert requests.get(url).content == requests.get(url).content

tests/integration/test_wild.py

@@ -3,6 +3,8 @@ requests = pytest.importorskip("requests")
 
 import vcr
 
+import httplib
+
 
 def test_domain_redirect():
     '''Ensure that redirects across domains are considered unique'''
@@ -15,3 +17,37 @@ def test_domain_redirect():
         # Ensure that we've now served two responses. One for the original
         # redirect, and a second for the actual fetch
         assert len(cass) == 2
+
+
+def test_flickr_multipart_upload():
+    """
+    The python-flickr-api project does a multipart
+    upload that confuses vcrpy
+    """
+    def _pretend_to_be_flickr_library():
+        content_type, body = "text/plain", "HELLO WORLD"
+        h = httplib.HTTPConnection("httpbin.org")
+        headers = {
+            "Content-Type": content_type,
+            "content-length": str(len(body))
+        }
+        h.request("POST", "/post/", headers=headers)
+        h.send(body)
+        r = h.getresponse()
+        data = r.read()
+        h.close()
+
+    with vcr.use_cassette('fixtures/vcr_cassettes/flickr.json') as cass:
+        _pretend_to_be_flickr_library()
+        assert len(cass) == 1
+
+    with vcr.use_cassette('fixtures/vcr_cassettes/flickr.json') as cass:
+        assert len(cass) == 1
+        _pretend_to_be_flickr_library()
+        assert cass.play_count == 1
+
+def test_flickr_should_respond_with_200(tmpdir):
+    testfile = str(tmpdir.join('flickr.yml'))
+    with vcr.use_cassette(testfile):
+        r = requests.post("http://api.flickr.com/services/upload")
+        assert r.status_code == 200

tests/unit/test_cassettes.py

@@ -1,5 +1,6 @@
 import pytest
 import yaml
+import mock
 from vcr.cassette import Cassette
 
 
@@ -46,18 +47,24 @@ def test_cassette_len():
     assert len(a) == 2
 
 
+def _mock_requests_match(request1, request2, matchers):
+    return request1 == request2
+
+@mock.patch('vcr.cassette.requests_match', _mock_requests_match)
 def test_cassette_contains():
     a = Cassette('test')
     a.append('foo', 'bar')
     assert 'foo' in a
 
 
+@mock.patch('vcr.cassette.requests_match', _mock_requests_match)
 def test_cassette_response_of():
     a = Cassette('test')
     a.append('foo', 'bar')
     assert a.response_of('foo') == 'bar'
 
 
+@mock.patch('vcr.cassette.requests_match', _mock_requests_match)
 def test_cassette_get_missing_response():
     a = Cassette('test')
     with pytest.raises(KeyError):

vcr/cassette.py

@@ -10,10 +10,12 @@ except ImportError:
 from .patch import install, reset
 from .persist import load_cassette, save_cassette
 from .serializers import yamlserializer
+from .matchers import requests_match, url, method
 
 
 class Cassette(object):
     '''A container for recorded requests and responses'''
+
     @classmethod
     def load(cls, path, **kwargs):
         '''Load in the cassette stored at the provided path'''
@@ -21,12 +23,20 @@ class Cassette(object):
         new_cassette._load()
         return new_cassette
 
-    def __init__(self, path, serializer=yamlserializer):
+    def __init__(self,
+                 path,
+                 serializer=yamlserializer,
+                 record_mode='once',
+                 match_on=[url, method]):
         self._path = path
         self._serializer = serializer
-        self.data = OrderedDict()
+        self._match_on = match_on
+
+        # self.data is the list of (req, resp) tuples
+        self.data = []
         self.play_counts = Counter()
         self.dirty = False
+        self.record_mode = record_mode
 
     @property
     def play_count(self):
@@ -34,11 +44,25 @@ class Cassette(object):
 
     @property
     def requests(self):
-        return self.data.keys()
+        return [request for (request, response) in self.data]
 
     @property
     def responses(self):
-        return self.data.values()
+        return [response for (request, response) in self.data]
+
+    @property
+    def rewound(self):
+        """
+        If the cassette has already been recorded in another session, and has
+        been loaded again fresh from disk, it has been "rewound".  This means
+        that it should be write-only, depending on the record mode specified
+        """
+        return not self.dirty and self.play_count
+
+    @property
+    def write_protected(self):
+        return self.rewound and self.record_mode == 'once' or \
+            self.record_mode == 'none'
 
     def mark_played(self, request):
         '''
@@ -48,12 +72,25 @@ class Cassette(object):
 
     def append(self, request, response):
         '''Add a request, response pair to this cassette'''
-        self.data[request] = response
+        self.data.append((request, response))
         self.dirty = True
 
     def response_of(self, request):
-        '''Find the response corresponding to a request'''
-        return self.data[request]
+        '''
+        Find the response corresponding to a request
+
+        '''
+        responses = []
+        for stored_request, response in self.data:
+            if requests_match(request, stored_request, self._match_on):
+                responses.append(response)
+        index = self.play_counts[request]
+        try:
+            return responses[index]
+        except IndexError:
+            # I decided that a KeyError is the best exception to raise
+            # if the cassette doesn't contain the request asked for.
+            raise KeyError
 
     def _as_dict(self):
         return {"requests": self.requests, "responses": self.responses}
@@ -90,7 +127,10 @@ class Cassette(object):
 
     def __contains__(self, request):
         '''Return whether or not a request has been stored'''
-        return request in self.data
+        for stored_request, response in self.data:
+            if requests_match(stored_request, request, self._match_on):
+                return True
+        return False
 
     def __enter__(self):
         '''Patch the fetching libraries we know about'''

vcr/config.py

@@ -1,16 +1,30 @@
 import os
 from .cassette import Cassette
 from .serializers import yamlserializer, jsonserializer
+from .matchers import method, url, host, path, headers, body
 
 
 class VCR(object):
-    def __init__(self, serializer='yaml', cassette_library_dir=None):
+    def __init__(self,
+                 serializer='yaml',
+                 cassette_library_dir=None,
+                 record_mode="once"):
         self.serializer = serializer
+        self.match_on = ['url', 'method']
         self.cassette_library_dir = cassette_library_dir
         self.serializers = {
             'yaml': yamlserializer,
             'json': jsonserializer,
         }
+        self.matchers = {
+            'method': method,
+            'url': url,
+            'host': host,
+            'path': path,
+            'headers': headers,
+            'body': body,
+        }
+        self.record_mode = record_mode
 
     def _get_serializer(self, serializer_name):
         try:
@@ -22,8 +36,19 @@ class VCR(object):
             raise KeyError
         return serializer
 
+    def _get_matchers(self, matcher_names):
+        try:
+            matchers = [self.matchers[m] for m in matcher_names]
+        except KeyError:
+            print "Matcher {0} doesn't exist or isn't registered".format(
+                matcher_name
+            )
+            raise KeyError
+        return matchers
+
     def use_cassette(self, path, **kwargs):
         serializer_name = kwargs.get('serializer', self.serializer)
+        matcher_names = kwargs.get('match_on', self.match_on)
         cassette_library_dir = kwargs.get(
             'cassette_library_dir',
             self.cassette_library_dir
@@ -34,9 +59,14 @@ class VCR(object):
 
         merged_config = {
             "serializer": self._get_serializer(serializer_name),
+            "match_on": self._get_matchers(matcher_names),
+            "record_mode": kwargs.get('record_mode', self.record_mode),
         }
 
         return Cassette.load(path, **merged_config)
 
     def register_serializer(self, name, serializer):
         self.serializers[name] = serializer
+
+    def register_matcher(self, name, matcher):
+        self.matchers[name] = matcher

vcr/matchers.py

@@ -0,0 +1,26 @@
+def method(r1, r2):
+    return r1.method == r2.method
+
+
+def url(r1, r2):
+    return r1.url == r2.url
+
+
+def host(r1, r2):
+    return r1.host == r2.host
+
+
+def path(r1, r2):
+    return r1.path == r2.path
+
+
+def body(r1, r2):
+    return r1.body == r2.body
+
+
+def headers(r1, r2):
+    return r1.headers == r2.headers
+
+
+def requests_match(r1, r2, matchers):
+    return all(m(r1, r2) for m in matchers)

vcr/patch.py

@@ -25,7 +25,11 @@ except ImportError:  # pragma: no cover
 
 
 def install(cassette):
-    '''Install a cassette in lieu of actuall fetching'''
+    """
+    Patch all the HTTPConnections references we can find!
+    This replaces the actual HTTPConnection with a VCRHTTPConnection
+    object which knows how to save to / read from cassettes
+    """
     httplib.HTTPConnection = httplib.HTTP._connection_class = VCRHTTPConnection
     httplib.HTTPSConnection = httplib.HTTPS._connection_class = (
         VCRHTTPSConnection)

vcr/stubs/__init__.py

@@ -64,29 +64,111 @@ class VCRConnectionMixin:
             headers=headers or {}
         )
 
-        # Check if we have a cassette set, and if we have a response saved.
-        # If so, there's no need to keep processing and we can bail
-        if self.cassette and self._vcr_request in self.cassette:
-            return
+        # Note: The request may not actually be finished at this point, so
+        # I'm not sending the actual request until getresponse().  This
+        # allows me to compare the entire length of the response to see if it
+        # exists in the cassette.
 
-        # Otherwise, we should submit the request
-        self._baseclass.request(
-            self, method, url, body=body, headers=headers or {})
+    def send(self, data):
+        '''
+        This method is called after request(), to add additional data to the
+        body of the request.  So if that happens, let's just append the data
+        onto the most recent request in the cassette.
+        '''
+        self._vcr_request.body = (self._vcr_request.body or '') + data
+
+    def _send_request(self, method, url, body, headers):
+        """
+        Copy+pasted from python stdlib 2.6 source because it
+        has a call to self.send() which I have overridden
+        #stdlibproblems #fml
+        """
+        header_names = dict.fromkeys([k.lower() for k in headers])
+        skips = {}
+        if 'host' in header_names:
+            skips['skip_host'] = 1
+        if 'accept-encoding' in header_names:
+            skips['skip_accept_encoding'] = 1
+
+        self.putrequest(method, url, **skips)
+
+        if body and ('content-length' not in header_names):
+            thelen = None
+            try:
+                thelen = str(len(body))
+            except TypeError, te:
+                # If this is a file-like object, try to
+                # fstat its file descriptor
+                import os
+                try:
+                    thelen = str(os.fstat(body.fileno()).st_size)
+                except (AttributeError, OSError):
+                    # Don't send a length if this failed
+                    if self.debuglevel > 0:
+                        print "Cannot stat!!"
+
+            if thelen is not None:
+                self.putheader('Content-Length', thelen)
+        for hdr, value in headers.iteritems():
+            self.putheader(hdr, value)
+        self.endheaders()
+
+        if body:
+            self._baseclass.send(self, body)
+
+    def _send_output(self, message_body=None):
+        """
+        Copy-and-pasted from httplib, just so I can modify the self.send()
+        calls to call the superclass's send(), since I had to override the
+        send() behavior, since send() is both an external and internal
+        httplib API.
+        """
+        self._buffer.extend(("", ""))
+        msg = "\r\n".join(self._buffer)
+        del self._buffer[:]
+        # If msg and message_body are sent in a single send() call,
+        # it will avoid performance problems caused by the interaction
+        # between delayed ack and the Nagle algorithm.
+        if isinstance(message_body, str):
+            msg += message_body
+            message_body = None
+        self._baseclass.send(self, msg)
+        if message_body is not None:
+            #message_body was not a string (i.e. it is a file) and
+            #we must run the risk of Nagle
+            self._baseclass.send(self, message_body)
 
     def getresponse(self, _=False):
         '''Retrieve a the response'''
         # Check to see if the cassette has a response for this request. If so,
         # then return it
-        if self._vcr_request in self.cassette:
+        if self._vcr_request in self.cassette and \
+                self.cassette.record_mode != "all":
             response = self.cassette.response_of(self._vcr_request)
             # Alert the cassette to the fact that we've served another
             # response for the provided requests
             self.cassette.mark_played(self._vcr_request)
             return VCRHTTPResponse(response)
         else:
-            # Otherwise, we made an actual request, and should return the
-            # response we got from the actual connection
-            response = HTTPConnection.getresponse(self)
+            if self.cassette.write_protected:
+                raise Exception("cassette is write protected")
+
+            # Otherwise, we should send the request, then get the response
+            # and return it.
+
+            # make the request
+            self._baseclass.request(
+                self,
+                method=self._vcr_request.method,
+                url=self._vcr_request.path,
+                body=self._vcr_request.body,
+                headers=dict(self._vcr_request.headers or {})
+            )
+
+            # get the response
+            response = self._baseclass.getresponse(self)
+
+            # put the response into the cassette
             response = {
                 'status': {
                     'code': response.status,