added a few more features to qc

.travis.yml

@@ -0,0 +1,7 @@
+language: python
+python:
+  - "2.6"
+  - "2.7"
+  - "pypy"
+install: "pip install --requirement requirements-test.txt --use-mirrors"
+script: nosetests

README.rst

@@ -0,0 +1,139 @@
+============
+ QuickCheck
+============
+
+.. image:: https://travis-ci.org/davidedelvento/qc.png
+   :target: https://travis-ci.org/davidedelvento/qc
+
+Introduction
+============
+
+This framework does Random and Combinatorial Testing. It is a Python framework
+inspired by Haskell's QuickCheck_ and Scala's scalacheck_ (not a port to Python
+of those frameworks). The framework is not standalone, but works in your own
+favorite testing infrastructure: PyUnit (a.k.a. unittest_) nose_ and py.test_ are
+all supported.
+
+In Combinatorial Testing, `qc` provides a deterministic implementation of all-choices
+(simply picks all the combinatorial choices of parameters) and all-pairs_
+algorithms. All-choices simply tests all possible choice of input parameters, so it's
+an exhaustive, but very slow technique when there are more than a few parameters with
+more than a handful of possible values each. The all-pairs_ algorithm is very
+clever and exponentially reduces the running time, while still guaranteeing that each
+pair of input parameter is tested for each possible combination of values.
+
+In Random Testing, `qc` provides many convenient ways of generating test cases, and
+a very useful automatic test case reduction. The test case reduction uses binary search
+to find a small, failing test case, very quickly.
+
+.. _QuickCheck: http://hackage.haskell.org/package/QuickCheck
+.. _scalacheck: https://github.com/rickynils/scalacheck
+.. _all-pairs: https://en.wikipedia.org/wiki/All-pairs_testing
+.. _unittest: https://docs.python.org/2/library/unittest.html
+.. _nose: https://nose.readthedocs.org/en/latest/
+.. _py.test: http://pytest.org/latest/
+
+More on Random Testing
+======================
+
+Since Random Testing is not popular, you may want to
+have a look at the following videos from Professor
+John Regehr, University of Utah (less than 15 minutes total) if you are not
+familiar with the technique:
+
+* `Introduction to random testing <http://www.youtube.com/watch?v=cwhC19Fa_84>`_
+* `Why random testing is good (1) <http://www.youtube.com/watch?v=PrJZ6144eeM>`_
+* `Why random testing is good (2) <http://www.youtube.com/watch?v=btlfWwyzSXQ>`_
+* `Why random testing is good (3) <http://www.youtube.com/watch?v=iw6BtJxPT8A>`_
+* `Why random testing is good (4) <http://www.youtube.com/watch?v=QrLtkSdMDgw>`_
+
+As you can see, Random Testing is not just randomly feeding your software a random
+stream of bytes. It requires more thoughts. The `Udacity course on
+testing`_ has units 3 and 4 entirely dedicated to Random Testing,
+describing many things you need to know about Random Testing: how to
+create valid, good, random test cases (3.25-26 and 4.5-15), mutators
+(3.29), oracles (3.30-34), test case reduction (4.5-6), tradeoffs
+(4.18-20), and more.  Random Testing is also mentioned elsewhere in
+the class (and expected to be known in the final exam). It is very
+worth watching (the introductory videos linked above are from this
+class).
+
+.. _Udacity course on testing: http://www.udacity.com/overview/Course/cs258/CourseRev/1
+
+Installation
+============
+
+The easy, system-wide way (requires administrative privileges)::
+
+    sudo pip install -e git://github.com/davidedelvento/qc.git#egg=qc
+
+If you don't feel ready to commit for a whole system install of this library, or
+simply don't have root access on your machine (and don't want to use virtualenv_), 
+just copy the ``qc`` directory 
+and its content (as seen in https://github.com/davidedelvento/qc/tree/master/qc
+at the moment the content is a mere ``__init__.py`` file) into the location of your choice.  
+To make `qc` available to your programs you will have to set the
+``PYTHONPATH`` environmental variable or have the ``qc`` directory as
+a subdirectory of the tree where you are running (for details, see 
+https://docs.python.org/2/tutorial/modules.html#the-module-search-path )
+
+.. _virtualenv: http://virtualenv.readthedocs.org/en/latest/virtualenv.html
+
+
+Examples
+========
+
+These examples are more to be read than to be run, but of course you want to
+run them to see the framework in action (and of course all the failures are
+there on purpose...)
+
+``examples/ex1_unittest.py`` and ``examples/ex1_nose.py``
+    These files provide a simple example (borrowed from scalacheck)
+    on how to use this framework with Python native PyUnit framework
+    (aka unittest module) and with the popular nose framework.  Just
+    run ``python examples/example1_unittest.py`` or ``nosetests
+    examples/example1_nose.py``. Py.test can run both the nose test or the unittest
+    using ``py.test examples/example1_nose.py`` or ``py.test examples/ex1_nose.py``
+
+``examples/ex2_choices_pairs.py`` TBD
+
+``examples/ex3_airplane.py``
+    Simple example on how to have qc generate your own custom (random)
+    objects and how to use them in practice. Run it with either
+    ``nosetests examples/ex3_airplane.py`` or ``py.test examples/ex3_airplane.py``
+
+``examples/ex4_bookcase.py``
+    A more elaborate example, showing the power of automatic shrinking
+    and showing how to write your own shrinker. In this example you can
+    see how qc automagically finds the root cause of the bug, compare
+    the output with or without shrinking! Run it with either
+    ``nosetests examples/ex4_bookcase.py`` or ``py.test examples/ex4_bookcase.py``
+
+
+
+Known bugs
+==========
+
+See https://github.com/davidedelvento/qc/issues for a list of known
+issues. 
+
+One common problem when using automatic shrinking is running out
+of stack space in the recursion process (the shrink algorithm call
+itself several times to produce a smaller test case). This may happen
+either if there is a bug in qc itself, or if there is a problem in
+your test code. You will see an error like::
+
+    RuntimeError: maximum recursion depth exceeded while calling a Python object
+
+with a stack trace that shows the recursion tree of the shrinking
+method calling itself. To understand what is happening, it is usually
+useful to add the ``shrink=False`` option to the ``@forall`` decorator
+of the affected test method. In very rare cases it may be necessary to
+increase the stack depth with a call to
+``sys.setrecursionlimit(NEWDEPTH)``, but do not do it until you
+understand that it is really the case for your test. More often than
+not, there will be a bug in your code (especially likely if you are
+writing your first shrinker) or in qc.  Please report the
+latter to https://github.com/davidedelvento/qc/issues
+
+

examples/ex1_nose.py

@@ -0,0 +1,27 @@
+from qc import forall, unicodes
+
+# This example is adapted from Scala's
+# https://github.com/rickynils/scalacheck
+# and we are pretending to test the string
+# concatenation, slicing and the len builtin
+
+@forall(tries=2000, a=unicodes(), b=unicodes())
+def testStartswith(a, b):
+    concat = a + b
+    assert concat.startswith(a)
+
+@forall(tries=10, a=unicodes(), b=unicodes())
+def testConcatenation(a, b):
+    concat = a + b
+    # the following is meant to fail as an example of a failure
+    assert len(concat) > len(a)
+    assert len(concat) > len(b)
+
+@forall(a=unicodes(), b=unicodes(), c=unicodes())
+def testSubstring(a, b, c):
+    concat = a + b + c
+    start = len(a)
+    stop = len(a) + len(b)
+    assert concat[start: stop] == b
+
+

examples/ex1_unittest.py

@@ -0,0 +1,31 @@
+from unittest import TestCase, main
+from qc import forall, unicodes
+
+# This example is adapted from Scala's
+# https://github.com/rickynils/scalacheck
+# and we are pretending to test the string
+# concatenation, slicing and the len builtin
+
+class TestString(TestCase):
+    @forall(tries=2000, a=unicodes(), b=unicodes())
+    def testStartswith(self, a, b):
+        concat = a + b
+        self.assertTrue(concat.startswith(a))
+
+    @forall(tries=10, a=unicodes(), b=unicodes())
+    def testConcatenation(self, a, b):
+        concat = a + b
+        # the following is meant to fail as an example of a failure
+        self.assertTrue(len(concat) > len(a))
+        self.assertTrue(len(concat) > len(b))
+
+    @forall(a=unicodes(), b=unicodes(), c=unicodes())
+    def testSubstring(self, a, b, c):
+        concat = a + b + c
+        start = len(a)
+        stop = len(a) + len(b)
+        self.assertEqual(concat[start: stop], b)
+
+if __name__ == "__main__":
+    main() 
+

examples/ex3_airplane.py

@@ -0,0 +1,61 @@
+# Location is a (lame) class needed by the
+# class under test. In real life it will be
+# a non-lame object you really need
+class Location(object):
+    def __init__(self, lat, lon, height):
+        self.lat=lat
+        self.lon=lon
+        self.h=height
+
+# This is the class under test
+class Plane(object):  
+    def __init__(self):
+        self.location = Location(46.22, -112.1, 4968)
+    def fly_to(self, loc):
+        if (-75 < loc.lon < -70 and
+             20 < loc.lat <  30 and
+            loc.h < 6000):
+            raise Exception("Plane has disappeared in the Bermuda triangle")
+        else:
+            self.location = loc
+    def has_gas(self):
+        return True
+
+from unittest import TestCase, main
+from qc import forall, floats
+
+# This is the qc-related auxiliary method
+# used to create random objects. In this
+# simple example it is overkill, but the
+# purpose here is just to show how it is
+# done: in real life you may have several
+# testing methods requiring random locations
+# and defining the method this way will
+# allow @forall to be able to inject 
+# Location's everywhere you need them
+def locations(lat = floats(-90,+90),
+              lon = floats(-180, +180),
+              height = floats(0, 30000)):
+    while True:
+        yield Location(lat.next(), lon.next(), height.next())
+
+
+# This is the PyUnit test case
+# In this example we are just
+# flying the plane to random locations
+# and assert that it does not run out
+# of gas. In real life you would assert
+# that it *does* run out of gas, but it
+# does *not* fall apart in other ways.
+class TestPlane(TestCase):
+    def setUp(self):
+        self.cessna172 = Plane()
+
+    @forall(tries=50000, l=locations())
+    def testFly(self, l):
+        self.cessna172.fly_to(l)
+        self.assertTrue(self.cessna172.has_gas())
+
+if __name__ == '__main__':
+    main()
+

examples/ex4_bookcase.py

@@ -0,0 +1,63 @@
+# This is at the same time the class under test,
+# and the class that you need to randomly generate
+# For simplicity a book is uniquely identified
+# by a just a positive integer
+class Bookcase(object):
+    def __init__(self, num_shelves, books):
+        self.num_shelves = num_shelves
+        self.books = books
+    def put(self, book):
+        if not self.full():
+            self.books.append(book)
+    def take(self, book):
+        if book in self.books:
+            self.books.remove(book)
+        if 13 in self.books: # artificially introduced bug
+            return 13
+        else:
+            return book
+    def full(self):
+        return False # This bookcase has infinite capacity
+    def __repr__(self):
+        return "Bookcase with " + str(self.num_shelves) + " shelves and containing books: " + self.books.__repr__()
+
+from unittest import TestCase, main
+from qc import forall, integers, lists, qc_shrink
+
+# This is the qc-related auxiliary method
+# used to create random objects (bookcases)
+def bookcases(nshelves = integers(1,10), book_set=lists(items=integers(1,20))):
+    while True:
+        yield Bookcase(nshelves.next(), book_set.next())
+
+# This is the custom shrinking function we have to provide
+# to qc. It has to decide the logic by which we want the
+# test case to be shrunk. In this case, we just shrink
+# the list of books, using qc default's shrink function.
+# Note that we are not shrinking the number of shelves.
+# This is the common pattern to create custom shrinker:
+# decide what has to change and what has not. Call
+# qc's shrinker on the relevant objects (if they are
+# supported, otherwise write your own shrinker for those)
+def shrink_bookcase(bookcase):
+    if isinstance(bookcase, Bookcase):
+        for shrinked_books in qc_shrink(bookcase.books):
+            yield Bookcase(bookcase.num_shelves, shrinked_books)
+    # else we do not shrink
+
+# This is the PyUnit test case
+class TestBookcase(TestCase):
+    @forall(bc=bookcases(), book=integers())
+    def testPutAndTake_noshrink(self, bc, book):
+        bc.put(book)
+        self.assertEqual(book, bc.take(book))
+
+    @forall(bc=bookcases(), book=integers(), custom_shrink=shrink_bookcase)
+    def testPutAndTake_shrink(self, bc, book):
+        bc.put(book)
+        self.assertEqual(book, bc.take(book))
+
+if __name__ == '__main__':
+    main()
+
+