Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
c085e81
In case of failure, print the whole object which caused it
Jul 20, 2012
b4af832
Initialized the random seed generator, either at random, or with user…
Jul 20, 2012
2fd6756
first, ugly and rudimentary, shrinking attempt
Jul 20, 2012
4f32ee8
Shrinking for a generic Exception would case ad infinitum recursion,
Jul 26, 2012
88769ff
Report of what happened.
Jul 26, 2012
8147d40
Added tests for shrink function, and implemented what was needed to
Aug 1, 2012
0a3243b
A decent README
Aug 23, 2012
8fd011b
If a length of subscriptable object is zero the previous shrink
Aug 23, 2012
a0926d4
first unittest example
Aug 23, 2012
fc3901a
Since there will be at least 4 examples, a directory has
Aug 23, 2012
a5323fc
Ported the first example to nose
Aug 23, 2012
4bd2bd0
Clarification on test case reduction in the README
Aug 23, 2012
dc41701
Example 3: simple one on how to have qc generate your own custom
Aug 24, 2012
826c307
A minimal code clean-up and a bunch of comments
Aug 24, 2012
950249d
In case of single-element collections (e.g. lists), there was
Aug 24, 2012
1aaa73c
The bookcase example draft
Aug 24, 2012
c1f2401
Now it is possible to specify a custom shrink function
Aug 24, 2012
f594e5c
Conclusions on the bookcase example
Aug 24, 2012
65fda0e
Simplified the name of two examples, and updated the README
Aug 27, 2012
4c66718
Added some utterance in the README about the possibility of
Aug 27, 2012
9635177
Add to README on running examples with pytest.
seanfisk Jun 5, 2013
a66a0d6
Reformat README as reStructuredText.
seanfisk Jun 5, 2013
11a3950
Add Travis-CI support.
seanfisk Jun 5, 2013
7dc673a
Merge pull request #1 from seanfisk/seanfisk/improve-readme
davidedelvento Jun 5, 2013
0842dd8
Merge pull request #2 from seanfisk/seanfisk/travis-ci
davidedelvento Jun 5, 2013
3b2ce08
Added Travis image to the README
davidedelvento Jun 6, 2013
1fbd07a
It looks like Dan Bravender is not anymore interested in maintaining …
Sep 18, 2014
9d59dda
clarified the verbiage about bugs
Sep 18, 2014
9e7a648
using a custom class instead of re-raising the previous exception, to…
Sep 18, 2014
6e67603
slowly introducing full Python3 syntax: here two exceptions
Sep 18, 2014
2530902
verbose enables the pretty-printing of the whole shrink history
Sep 19, 2014
d010f87
better default values for integers and their tests
Sep 19, 2014
4705e2f
better default values for floats and their tests
Sep 19, 2014
9cc81d5
do not shrink INFs
Sep 19, 2014
5a8a4f4
Lists and tuples tests
Sep 19, 2014
cb168ff
make sure correct behavior when requesting incorrect list/tuple size
Sep 19, 2014
ab9f8c2
moving the order of tests around
Sep 19, 2014
d823c93
comments about testing dicts
Sep 22, 2014
8856896
clarified some shrinking tests
Sep 22, 2014
3d1ed70
testing that all the cases of floats are covered -- and making sure t…
Sep 22, 2014
b99e4cf
clarified the shrinking of floats applies to large one -- comment on …
Sep 22, 2014
efdc392
This does not seem to be printed as desired, it gets repeated inappro…
Sep 26, 2014
3ecb3e1
some tests for the forall wrapper
Sep 26, 2014
a705d15
All choice combinatorial testing
Oct 3, 2014
ce52be7
suppressing the warning for the test, since the test case is small en…
Oct 6, 2014
dc5498e
TDD simple (binary-only) all pairs test and a dummy (test-failing) im…
Oct 6, 2014
a4d82f9
Updated and cleaned up the README
Oct 7, 2014
b7cfa4e
TDD: simple all pair implementation with very limited functionality
Oct 7, 2014
c5ed801
marking the fully all-pairs test as skipped instead of failed
Oct 7, 2014
8a90d3c
nicer, more generic implementation to generate all pairs for binary v…
Oct 9, 2014
a8f70e2
Removed the TODO section, moved to github issues
Oct 13, 2014
a45ec7f
Removed another reference to Python 3 (see issue#9 for details)
Oct 13, 2014
657a3c0
mentioning virtualenv in the README
Oct 14, 2014
4ed5e23
updating examples and README for any test framework
Oct 23, 2014
6052be0
Commit 253090290e1d45b0fb5d02bfe8aec2e74c03885f changed the default v…
Oct 23, 2014
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
language: python
python:
- "2.6"
- "2.7"
- "pypy"
install: "pip install --requirement requirements-test.txt --use-mirrors"
script: nosetests
6 changes: 0 additions & 6 deletions README

This file was deleted.

139 changes: 139 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
============
QuickCheck
============

.. image:: https://travis-ci.org/davidedelvento/qc.png
:target: https://travis-ci.org/davidedelvento/qc

Introduction
============

This framework does Random and Combinatorial Testing. It is a Python framework
inspired by Haskell's QuickCheck_ and Scala's scalacheck_ (not a port to Python
of those frameworks). The framework is not standalone, but works in your own
favorite testing infrastructure: PyUnit (a.k.a. unittest_) nose_ and py.test_ are
all supported.

In Combinatorial Testing, `qc` provides a deterministic implementation of all-choices
(simply picks all the combinatorial choices of parameters) and all-pairs_
algorithms. All-choices simply tests all possible choice of input parameters, so it's
an exhaustive, but very slow technique when there are more than a few parameters with
more than a handful of possible values each. The all-pairs_ algorithm is very
clever and exponentially reduces the running time, while still guaranteeing that each
pair of input parameter is tested for each possible combination of values.

In Random Testing, `qc` provides many convenient ways of generating test cases, and
a very useful automatic test case reduction. The test case reduction uses binary search
to find a small, failing test case, very quickly.

.. _QuickCheck: http://hackage.haskell.org/package/QuickCheck
.. _scalacheck: https://github.com/rickynils/scalacheck
.. _all-pairs: https://en.wikipedia.org/wiki/All-pairs_testing
.. _unittest: https://docs.python.org/2/library/unittest.html
.. _nose: https://nose.readthedocs.org/en/latest/
.. _py.test: http://pytest.org/latest/

More on Random Testing
======================

Since Random Testing is not popular, you may want to
have a look at the following videos from Professor
John Regehr, University of Utah (less than 15 minutes total) if you are not
familiar with the technique:

* `Introduction to random testing <http://www.youtube.com/watch?v=cwhC19Fa_84>`_
* `Why random testing is good (1) <http://www.youtube.com/watch?v=PrJZ6144eeM>`_
* `Why random testing is good (2) <http://www.youtube.com/watch?v=btlfWwyzSXQ>`_
* `Why random testing is good (3) <http://www.youtube.com/watch?v=iw6BtJxPT8A>`_
* `Why random testing is good (4) <http://www.youtube.com/watch?v=QrLtkSdMDgw>`_

As you can see, Random Testing is not just randomly feeding your software a random
stream of bytes. It requires more thoughts. The `Udacity course on
testing`_ has units 3 and 4 entirely dedicated to Random Testing,
describing many things you need to know about Random Testing: how to
create valid, good, random test cases (3.25-26 and 4.5-15), mutators
(3.29), oracles (3.30-34), test case reduction (4.5-6), tradeoffs
(4.18-20), and more. Random Testing is also mentioned elsewhere in
the class (and expected to be known in the final exam). It is very
worth watching (the introductory videos linked above are from this
class).

.. _Udacity course on testing: http://www.udacity.com/overview/Course/cs258/CourseRev/1

Installation
============

The easy, system-wide way (requires administrative privileges)::

sudo pip install -e git://github.com/davidedelvento/qc.git#egg=qc

If you don't feel ready to commit for a whole system install of this library, or
simply don't have root access on your machine (and don't want to use virtualenv_),
just copy the ``qc`` directory
and its content (as seen in https://github.com/davidedelvento/qc/tree/master/qc
at the moment the content is a mere ``__init__.py`` file) into the location of your choice.
To make `qc` available to your programs you will have to set the
``PYTHONPATH`` environmental variable or have the ``qc`` directory as
a subdirectory of the tree where you are running (for details, see
https://docs.python.org/2/tutorial/modules.html#the-module-search-path )

.. _virtualenv: http://virtualenv.readthedocs.org/en/latest/virtualenv.html


Examples
========

These examples are more to be read than to be run, but of course you want to
run them to see the framework in action (and of course all the failures are
there on purpose...)

``examples/ex1_unittest.py`` and ``examples/ex1_nose.py``
These files provide a simple example (borrowed from scalacheck)
on how to use this framework with Python native PyUnit framework
(aka unittest module) and with the popular nose framework. Just
run ``python examples/example1_unittest.py`` or ``nosetests
examples/example1_nose.py``. Py.test can run both the nose test or the unittest
using ``py.test examples/example1_nose.py`` or ``py.test examples/ex1_nose.py``

``examples/ex2_choices_pairs.py`` TBD

``examples/ex3_airplane.py``
Simple example on how to have qc generate your own custom (random)
objects and how to use them in practice. Run it with either
``nosetests examples/ex3_airplane.py`` or ``py.test examples/ex3_airplane.py``

``examples/ex4_bookcase.py``
A more elaborate example, showing the power of automatic shrinking
and showing how to write your own shrinker. In this example you can
see how qc automagically finds the root cause of the bug, compare
the output with or without shrinking! Run it with either
``nosetests examples/ex4_bookcase.py`` or ``py.test examples/ex4_bookcase.py``



Known bugs
==========

See https://github.com/davidedelvento/qc/issues for a list of known
issues.

One common problem when using automatic shrinking is running out
of stack space in the recursion process (the shrink algorithm call
itself several times to produce a smaller test case). This may happen
either if there is a bug in qc itself, or if there is a problem in
your test code. You will see an error like::

RuntimeError: maximum recursion depth exceeded while calling a Python object

with a stack trace that shows the recursion tree of the shrinking
method calling itself. To understand what is happening, it is usually
useful to add the ``shrink=False`` option to the ``@forall`` decorator
of the affected test method. In very rare cases it may be necessary to
increase the stack depth with a call to
``sys.setrecursionlimit(NEWDEPTH)``, but do not do it until you
understand that it is really the case for your test. More often than
not, there will be a bug in your code (especially likely if you are
writing your first shrinker) or in qc. Please report the
latter to https://github.com/davidedelvento/qc/issues


27 changes: 27 additions & 0 deletions examples/ex1_nose.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
from qc import forall, unicodes

# This example is adapted from Scala's
# https://github.com/rickynils/scalacheck
# and we are pretending to test the string
# concatenation, slicing and the len builtin

@forall(tries=2000, a=unicodes(), b=unicodes())
def testStartswith(a, b):
concat = a + b
assert concat.startswith(a)

@forall(tries=10, a=unicodes(), b=unicodes())
def testConcatenation(a, b):
concat = a + b
# the following is meant to fail as an example of a failure
assert len(concat) > len(a)
assert len(concat) > len(b)

@forall(a=unicodes(), b=unicodes(), c=unicodes())
def testSubstring(a, b, c):
concat = a + b + c
start = len(a)
stop = len(a) + len(b)
assert concat[start: stop] == b


31 changes: 31 additions & 0 deletions examples/ex1_unittest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
from unittest import TestCase, main
from qc import forall, unicodes

# This example is adapted from Scala's
# https://github.com/rickynils/scalacheck
# and we are pretending to test the string
# concatenation, slicing and the len builtin

class TestString(TestCase):
@forall(tries=2000, a=unicodes(), b=unicodes())
def testStartswith(self, a, b):
concat = a + b
self.assertTrue(concat.startswith(a))

@forall(tries=10, a=unicodes(), b=unicodes())
def testConcatenation(self, a, b):
concat = a + b
# the following is meant to fail as an example of a failure
self.assertTrue(len(concat) > len(a))
self.assertTrue(len(concat) > len(b))

@forall(a=unicodes(), b=unicodes(), c=unicodes())
def testSubstring(self, a, b, c):
concat = a + b + c
start = len(a)
stop = len(a) + len(b)
self.assertEqual(concat[start: stop], b)

if __name__ == "__main__":
main()

61 changes: 61 additions & 0 deletions examples/ex3_airplane.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Location is a (lame) class needed by the
# class under test. In real life it will be
# a non-lame object you really need
class Location(object):
def __init__(self, lat, lon, height):
self.lat=lat
self.lon=lon
self.h=height

# This is the class under test
class Plane(object):
def __init__(self):
self.location = Location(46.22, -112.1, 4968)
def fly_to(self, loc):
if (-75 < loc.lon < -70 and
20 < loc.lat < 30 and
loc.h < 6000):
raise Exception("Plane has disappeared in the Bermuda triangle")
else:
self.location = loc
def has_gas(self):
return True

from unittest import TestCase, main
from qc import forall, floats

# This is the qc-related auxiliary method
# used to create random objects. In this
# simple example it is overkill, but the
# purpose here is just to show how it is
# done: in real life you may have several
# testing methods requiring random locations
# and defining the method this way will
# allow @forall to be able to inject
# Location's everywhere you need them
def locations(lat = floats(-90,+90),
lon = floats(-180, +180),
height = floats(0, 30000)):
while True:
yield Location(lat.next(), lon.next(), height.next())


# This is the PyUnit test case
# In this example we are just
# flying the plane to random locations
# and assert that it does not run out
# of gas. In real life you would assert
# that it *does* run out of gas, but it
# does *not* fall apart in other ways.
class TestPlane(TestCase):
def setUp(self):
self.cessna172 = Plane()

@forall(tries=50000, l=locations())
def testFly(self, l):
self.cessna172.fly_to(l)
self.assertTrue(self.cessna172.has_gas())

if __name__ == '__main__':
main()

63 changes: 63 additions & 0 deletions examples/ex4_bookcase.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# This is at the same time the class under test,
# and the class that you need to randomly generate
# For simplicity a book is uniquely identified
# by a just a positive integer
class Bookcase(object):
def __init__(self, num_shelves, books):
self.num_shelves = num_shelves
self.books = books
def put(self, book):
if not self.full():
self.books.append(book)
def take(self, book):
if book in self.books:
self.books.remove(book)
if 13 in self.books: # artificially introduced bug
return 13
else:
return book
def full(self):
return False # This bookcase has infinite capacity
def __repr__(self):
return "Bookcase with " + str(self.num_shelves) + " shelves and containing books: " + self.books.__repr__()

from unittest import TestCase, main
from qc import forall, integers, lists, qc_shrink

# This is the qc-related auxiliary method
# used to create random objects (bookcases)
def bookcases(nshelves = integers(1,10), book_set=lists(items=integers(1,20))):
while True:
yield Bookcase(nshelves.next(), book_set.next())

# This is the custom shrinking function we have to provide
# to qc. It has to decide the logic by which we want the
# test case to be shrunk. In this case, we just shrink
# the list of books, using qc default's shrink function.
# Note that we are not shrinking the number of shelves.
# This is the common pattern to create custom shrinker:
# decide what has to change and what has not. Call
# qc's shrinker on the relevant objects (if they are
# supported, otherwise write your own shrinker for those)
def shrink_bookcase(bookcase):
if isinstance(bookcase, Bookcase):
for shrinked_books in qc_shrink(bookcase.books):
yield Bookcase(bookcase.num_shelves, shrinked_books)
# else we do not shrink

# This is the PyUnit test case
class TestBookcase(TestCase):
@forall(bc=bookcases(), book=integers())
def testPutAndTake_noshrink(self, bc, book):
bc.put(book)
self.assertEqual(book, bc.take(book))

@forall(bc=bookcases(), book=integers(), custom_shrink=shrink_bookcase)
def testPutAndTake_shrink(self, bc, book):
bc.put(book)
self.assertEqual(book, bc.take(book))

if __name__ == '__main__':
main()


Loading