Doctest Puns

I still have no idea why are unitests more useful. It takes more time to write and has pretty much the same result as a doctest. Is there a simple example of how unitesting a code produces more informative results?

Thanks to @teryror, support for running doctests has recently landed in Miri. So finally cargo miri test is executing all the same tests that cargo test runs. This has been a long-standing open issue and I am stoked that it is now finally resolved. :)

The Miri submodule in rustc has been updated, so doctest support will appear in the rustup-distributed Miri with the next nightly release. If that causes trouble for you, e.g. because Miri actually complains about some of your doctests, you can use cargo miri test --all-targets to run the other tests but not the doctests. If anything seems wrong, please report an issue. If you don't know what Miri is, our readme should help.

Next up: going over all the failing doctests in the standard library, and fixing them...

byexample is a tool that I made to overcome the shortcomings of Python's doctest. It allows you to execute snippets of code inside your documentation (blog post, tutorial, book, ...). In this way you can literally "execute" your docs and verify that they still up to date.

Around 2016 I was working on my final project (thesis-like) in Python and Javascript. I used doctest for testing/documenting Python but I couldn't find anything for Javascript and that how byexample born.

Now it supports snippets written in Ruby, Go, C++ and a few others languages but Python is the one with the best support (of course!).

It has a few years of development now but this the first time that I'm posting it in Reddit. Any feedback is welcome both here and in the Github project: https://github.com/byexamples/byexample

If you don't know what doctest is or you know but you it turned to be hard-to-debug/use, give byexample a try: at the end I created it to solve those issues!!

Thanks for your time!

https://i.redd.it/bch8v5c464h71.gif

I would like to write non-rust code (essentially ASCII art) into some of my function documentation, but when I do this by either indenting the "code" block or surrounding it with triple backticks, cargo test complains that a bunch of my "tests" are failing:

/// The following is not supposed to be a doctest for `f`:
///
///     I am not a doctest, but a formal grammar for a language
///     (or something of the sort).
///
/// Oh, but it is. 😈
fn f () {}

How does one prevent literal blocks in doc comments from being interpreted as doctests?

On OS-X I can't run doctests over real code. I've added a custom-setup and test-suite inside the .cabal file, a custom Setup.hs file, and a generic test/doctest.hs file. When running (cabal build doctest ; .../doctest) the linker hates things:

GHC runtime linker: fatal error: I found a duplicate definition for symbol
   __ZN17double_conversion7BitCastIdyEET_RKT0_
whilst processing object file
   /Users/tommd/.cabal/store/ghc-8.10.1/dbl-cnvrsn-2.0.2.0-d1095495/lib/libHSdbl-cnvrsn-2.0.2.0-d1095495.a
The symbol was previously defined in
   /Users/tommd/.cabal/store/ghc-8.10.1/dbl-cnvrsn-2.0.2.0-d1095495/lib/libHSdbl-cnvrsn-2.0.2.0-d1095495.a(double-conversion.o)
This could be caused by:
   * Loading two different object files which export the same symbol
   * Specifying the same object file twice on the GHCi command line
   * An incorrect `package.conf' entry, causing some object to be
     loaded twice.
doctests: doctests: unable to load package `double-conversion-2.0.2.0'

Is this a common issue that is understood? I wasn't successful googling it last time I tried.

I'm really... >> REALLY << excited to announce version 2.3 which brings an extensible reporter system to doctest - the fastest C++ testing framework! There are 2 reporters which come with the framework by default:

• console - normal output to stdout (can be redirected with --out=<filename>)
• XML - specific format to the framework (example)

This has been a 10+ month saga since I began refactoring doctest so the framework could have a decent reporter system. Since then a lot of spaghetti code was untangled and the framework became more maintainable (+ moving to C++11) - without breaking the interface for users.

XML output is vital for testing frameworks so they can be integrated properly in Continuous Integration (CI) workflows and has been a showstopper for a number of organizations in adopting doctest.

Some more information:

• The reporter system can also be used for implementing "event listeners" by users - just implement the IReporter interface and you can "listen" for events such as "test case starting/ending".
• multiple reporters can be used at the same time and the order they are called is based on their priority (specified when registering) - this way you could have a few listeners and 1...2+ reporters running at the same time (just be careful not to output to stdout from more than 1 reporter) - the set of reporters can be specified with --reporters=<filters> and all can be listed with --list-reporters
• xUnit/jUnit/TeamCity/compact reporters are coming in later versions (along with a ton of other things).

Once again

I've been trying out an implementation of doctests (as done in Python) for Emacs lisp code: https://github.com/riscy/doctest

These are two-liners that you insert in your documentation to illustrate how a function can be used, but which also double as simple unit tests. Does anyone know if something like this already exists for Emacs?

Python has doctests, which means you can write things in docstrings with a certain syntax and a test runner can treat them as assertions.

https://docs.python.org/3.7/library/doctest.html

• What do you think would need to change for lisp?
• What should they look like in lisp docstrings?
• What could we do in lisp that maybe python can't?
• Is it not a useful idea in lisp for some reason?

It is with great pleasure that I am announcing the release of Doctest 2.0 - the fastest feature-rich C++11 single-header testing framework for unit tests and TDD!

The main 4 developments are:

• moved to C++11 - motivated by the results from the poll in this reddit thread
• thread-safety - asserts and logging utilities can be used in multiple threads spawned from a single test case without race conditions (thread sanitizer tested) - see example
• a complete overhaul of the internals of the framework (moving from printf-style logging to streams and changing internal structures) to allow for easier future development (including a reporter interface - work in progress)
• given that doctest is extremely light on compile times and is meant to be used for tests side-by-side with production code - added the ability for asserts to be used outside of a testing context (as a general purpose assert library) - example

The move to C++11 allowed me to greatly simplify the codebase:

• removed a ton of preprocessor config options/detection for C++11 features (nullptr, deleted/defaulted functions, variadic macros, rvalue references, static assert, override, long long)
• using variadic templates where appropriate - mainly for templated test cases
• moved initialization of fields from initializer lists to class bodies
• using range-based for loops
• using lambdas
• using auto
• easily added portable thread-safety

Moving to C++11 also lead to droppin

I've seen the doctests documentation and even the PyCon 2020 YouTube channel has a video on doctests. However, I've never used doctests in my projects and I've never seen any GitHub or other projects that use doctests. What I usually see are various test frameworks like nose, pytest, or unittest.

So how common are doctests in reality? Have you ever used them? How beneficial are they compared to the various test frameworks?

Just migrated the unit tests for one of our projects from Boost.Test to Doctest and decided to share the time measurements while I still have the two branches at hand. You can see the number of the test cases in the run-time output below.

The tests were done on virtual machine with 4 cores and 8 GB of RAM.

The build was done with GCC 7.3 with -std='c++17' -O2 -Wall -Wextra -Werror -Winvalid-pch and 4 threads for parallel building. The setup with Doctest has two additional flags:

-DDOCTEST_CONFIG_SUPER_FAST_ASSERTS -DDOCTEST_CONFIG_NO_COMPARISON_WARNING_SUPPRESSION.

The Boost version that I'm using is 1.69 and the Doctest version is 2.3.2.

For the run-time tests, I ran them 3 times each and picked the slower ones. The timings for each version was in the same ballpark anyway.

Full rebuild, including the precompiled header, with Boost.Test

real    1m5.710s  
user    3m42.350s  
sys     0m5.751s

Full rebuild, including the precompiled header, with Doctest

real    0m52.567s  
user    2m40.018s  
sys     0m4.993s  

Rebuild, without the precompiled header, with Boost.Test

real    1m2.391s  
user    3m40.168s  
sys     0m5.532s  

Rebuild, without the precompiled header, with Doctest

real    0m46.351s  
user    2m38.131s  
sys     0m4.029s  

Run-time with Boost.Test

time ./tests.bin -r short  
Running 149 test cases...  
  
Test module "p3_tests" has passed with:  
  149 test cases out of 149 passed  
  1166 assertions out of 1166 passed  
  
real    0m0.056s  
user    0m0.025s  
sys     0m0.031s  

Run-time with Doctest

time ./tests.bin  
[doctest] doctest version is "2.3.2"  
[doctest] run with "--help" for options  
=============================================================================== 
[doctest] test cases:    149 |    149 passed |      0 failed |      0 skipped  
[doctest] assertions:   1166 |   1166 passed |      0 failed |  
[doctest] Status: SUCCESS!  

real    0m0.042s  
user    0m0.021s  
sys     0m0.021s

I love doctest.

Writing runnable examples in my documentation is a joy:

-- |
-- >>> loeb [ subtract 1 . (!!1), subtract 2 . (!!2), length ]
-- [0,1,3]
loeb :: Functor f => f (f a -> a) -> f a
loeb f = x where x = fmap ($x) f 

The one impedance point I run into sometime is that my example output is so big that it's not really legible.

data Rose = Rose [Rose] deriving Show

-- |
-- >>> rose 3
-- Rose [Rose [Rose [Rose []],Rose [Rose []]],Rose [Rose [Rose []],Rose [Rose []]],Rose [Rose [Rose []],Rose [Rose []]]]
rose :: Int -> Rose
rose n = Rose . replicate n . rose $ n - 1

I can make it more legible by adding some whitespace:

-- >>> rose 3
-- Rose 
--   [ Rose 
--     [ Rose
--       [ Rose []
--       ]
--     , Rose
--       [ Rose []
--       ]
--     ]
--   , Rose 
--     [ Rose
--       [ Rose []
--       ]
--     , Rose
--       [ Rose []
--       ]
--     ]
--   , Rose 
--     [ Rose
--       [ Rose []
--       ]
--     , Rose
--       [ Rose []
--       ]
--     ]
--   ]

But then doctest fails that example

### Failure in Example.hs:6: expression `rose 3'
expected: Rose 
            [ Rose 
              [ Rose
                [ Rose []
                ]
              , Rose
                [ Rose []
                ]
              ]
            , Rose 
              [ Rose
                [ Rose []
                ]
              , Rose
                [ Rose []
                ]
              ]
            , Rose 
              [ Rose
                [ Rose []
                ]
              , Rose
                [ Rose []
                ]
              ]
            ]
 but got: Rose [Rose [Rose [Rose []],Rose [Rose []]],Rose [Rose [Rose []],Rose [Rose []]],Rose [Rose [Rose []],Rose [Rose []]]]

doctest doesn't have a flag to make comparison of expected and actual example output whitespace-agnostic, so what's a coder to do?

One low-effort solution to this problem is to take advantage of the interpreter's :set -interactive-print option, which lets us choose an alternate printer.

For example, if I use the pretty-printer from pretty-show instead, it d

On 2016.05.22 I released version 1.0 of doctest. 1.1 brings the following major changes:

• HUGE improvements in compile time of asserts - 70-95% faster than the original one!

• wrote a FAQ - and the much requested differences with Catch section

• MANY minor fixes - it didn't compile when using nullptr in asserts, etc. - see the changelog

• improved documentation

• solved a very common problem of testing frameworks with self-registering tests inside static libraries - non-intrusive in pure CMake without any source code modification - see here

• added code coverage of the library

You can see the full generated changelog here

The framework is in a very stable condition but there are still a lot of features in the roadmap.

I truly believe this framework can become the preferred choice for testing in C++ - the foundations are solid and pure and I believe there is nothing else quite like it - see the main readme.

The project is in need of sponsors and publicity!

I don't want this post to be a cry for money, but I would greatly appreciate any financial support by individuals or companies using (or wanting to use/support) the framework.

I created a patreon page for that purpose in addition to the pledgie campaign.

You can checkout the hackernews thread for this here.

Any feedback is welcome!

EDIT:

[Baptiste Wich

What is your test system of choice and for what reasons? Bonus points i you can point to an "ELI5"-like tutorial on your system of choice. ;-)

For anyone else out there who likes to use vim when writing their haskell code, I want to take a minute to endorse the :make command.

By default it runs make in a subshell, but if you're averse to Makefiles, you can easily configure vim to run some other command, like cabal:

:set makeprg=cabal

I'll often set bindings to make recompiling, running or testing my code easy:

:let mapleader=" "
:nnoremap <Leader>b :make build<CR>
:nnoremap <Leader>r :make run<CR>
:nnoremap <Leader>t :make test<CR>

To me, the real benefit of :make over just doing :!make or :!cabal is the post-processing vim does on the output of makeprg, using 'errorformat' to parse the output and jump to the the file/line/column mentioned in the error message.

Compiler reports an error? Vim takes me right to where the error message says the problem is.

The default 'errorformat' handles error messages from cabal and ghc just fine, so there's no configuration required there.

When writing my haskell docs, I love using doctest to make sure my examples aren't lying. Doctest errors look a little different, though, and weren't getting successfully parsed for me.

For example, given two broken examples like:

-- Temp.hs
module Temp where

-- |
-- >>> foo "Unterminated string
-- "gnirts detanimretnU"
foo :: String -> String
foo = reverse    

-- |
-- >>> bar
-- "Hello"
bar :: String
bar = "World"

doctest Temp.hs reports:

### Failure in Temp.hs:4: expression `foo "Unterminated string'
expected: "\"gnirts detanimretnU\""
 but got: ""
          "\ESC[;1m<interactive>:23:25: \ESC[;1m\ESC[31merror:\ESC[0m\ESC[0m\ESC[;1m\ESC[0m\ESC[0m\ESC[;1m"
          "    lexical error in string/character literal at end of input\ESC[0m\ESC[0m"
          "\ESC[0m\ESC[0m\ESC[0m"
### Failure in Temp.hs:10: expression `bar'  
expected: "Hello"
 but got: "World"
Examples: 2  Tried: 2  Errors: 0  Failures: 2

vim would incorrectly parse out the error locations as:

• Line 4 in a file named "### Failure in Temp.hs"
• Column 25 of Line 23 in a file named "<interactive>"
• Line 10 of a file named "### Failure in Temp.hs"

To fix this, and enabl

I'm writing some docstrings for a Python, and some of them include examples in doctest format. I'm not using them as tests, just examples.

For those unfamiliar with the terminology, take a look at this:

def add(a, b):
    """Adds two numbers.
    
    Example:
    
    >>> add(2, 3)
    5
    >>> add(3,4)
    7
    """
    return a + b

The part with the >>> is the doctest part. It's meant to look exactly like a copy-pasted output of the python interpreter. The lines beginning with >>> are the code, and the lines following them are the results.

Is there a way (command or plugin) that updates/replaces those lines with the results?

For example, something that replaces:

    >>> add(2, 3)
    >>> add(3,4)

with

    >>> add(2, 3)
    5
    >>> add(3,4)
    7

and

    >>> add(2, 3)
    6
    >>> add(3,4)
    7

with

    >>> add(2, 3)
    5
    >>> add(3,4)
    7

I tried filtering selection encompassing the block through python, after removing all of the non-doctest lines with this command:

'<,'>!sed -e "/\s*>/\!d" -e "s/^\s*>>> //g" | python

but that only returns the results, like this:

5
7

Any ideas? I thought about starting a python interpreter instance in a tmux pane, piping the selection there and capturing the output, any better ideas (maybe without tmux)? Something with :terminal? A plugin?

For an assignment at school, we have to write a program which will print the day of the 10\10, 4\4 etc for each year, and submit doctests for it. My code is here: http://pastebin.com/kbShNP4P

But when I run doctest, I get this: doctest.testmod(Doomsday.doomsday(2012), verbose=True) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\python27\lib\doctest.py", line 1871, in testmod raise TypeError("testmod: module required; %r" % (m,)) TypeError: testmod: module required; 3

Please help, I can't seem to find any reason for this output even though the doctest command I used (doctest.testmod(Doomsday.doomsday(2012), verbose=True) was the same as my teacher's.

I'm happy to announce version 1.2 of doctest! The main changes are:

• improved runtime by more than 30 times compared to version 1.1.4 by eliminating allocations in the common case:
  • when asserts don't fail the expression is not stringified
  • rewrote the String class to employ the small string optimization
  • see the benchmarks
• templated test cases - parameterized by type
• an exception translation mechanism
• logging context with INFO("text: " << some_variable) with lazy string creation - only when an assert fails later in the current scope
• an API for reporting failures by third-party code such as mocking frameworks
• >> BREAKING CHANGE << the TEST_SUITE() macro now works with blocks of code
• multiple binaries (dll/exe) can share the same test runner/registry by defining the DOCTEST_CONFIG_IMPLEMENTATION_IN_DLL config identifier
• subcases can be filtered
• added support for un-parenthesized expressions containing commas in asserts - like this: CHECK(std::vector<int>{1, 2} == std::vector<int>{1, 3}); (only if variadic macros are enabled)
• crash handling support - using signals under UNIX and SEH under Windows - reports which test case failed and exits
• added support for test case decorators - description, skip, may_fail, should_fail, expected_failures, timeout, etc.
• added option to show duration of test case execution
• statically analysed on the CI - [Cppcheck](http://cppch

There are 2 new important things for doctest since the release of version 2.0:

• CHECK_THROWS_WITH(expr, exception_message); for testing if an expression threw an exception containing a specific message
• the DOCTEST_CONFIG_SUPER_FAST_ASSERTS config identifier now affects also the normal expression decomposing asserts such as CHECK(a == b); resulting in even faster compile times for asserts without having to rewrite them to the binary form of CHECK_EQ(a, b); - see the updated benchmarks

Release:

https://github.com/onqtam/doctest/releases/tag/2.2.0

Question: do you think DOCTEST_CONFIG_SUPER_FAST_ASSERTS should be enabled by default?

Or is there a plan or project implementing doctests in go like https://docs.python.org/3.7/library/doctest.html ?

How to run docttest examples using `stack test` rather than the doctest command?

Just migrated the unit tests for one of our projects from Boost.Test to Doctest and decided to share the time measurements while I still have the two branches at hand. You can see the number of the test cases in the run-time output below.

The tests were done on virtual machine with 4 cores and 8 GB of RAM.

The build was done with GCC 7.3 with -std='c++17' -O2 -Wall -Wextra -Werror -Winvalid-pch and 4 threads for parallel building. The setup with Doctest has two additional flags:

-DDOCTEST_CONFIG_SUPER_FAST_ASSERTS -DDOCTEST_CONFIG_NO_COMPARISON_WARNING_SUPPRESSION.

The Boost version that I'm using is 1.69 and the Doctest version is 2.3.2.

For the run-time tests, I ran them 3 times each and picked the slower ones. The timings for each version was in the same ballpark anyway.

Full rebuild, including the precompiled header, with Boost.Test

real    1m5.710s  
user    3m42.350s  
sys     0m5.751s

Full rebuild, including the precompiled header, with Doctest

real    0m52.567s  
user    2m40.018s  
sys     0m4.993s  

Rebuild, without the precompiled header, with Boost.Test

real    1m2.391s  
user    3m40.168s  
sys     0m5.532s  

Rebuild, without the precompiled header, with Doctest

real    0m46.351s  
user    2m38.131s  
sys     0m4.029s  

Run-time with Boost.Test

time ./tests.bin -r short  
Running 149 test cases...  
  
Test module "p3_tests" has passed with:  
  149 test cases out of 149 passed  
  1166 assertions out of 1166 passed  
  
real    0m0.056s  
user    0m0.025s  
sys     0m0.031s  

Run-time with Doctest

time ./tests.bin  
[doctest] doctest version is "2.3.2"  
[doctest] run with "--help" for options  
=============================================================================== 
[doctest] test cases:    149 |    149 passed |      0 failed |      0 skipped  
[doctest] assertions:   1167 |   1167 passed |      0 failed |  
[doctest] Status: SUCCESS!  

real    0m0.042s  
user    0m0.021s  
sys     0m0.021s

I released version 1.0 of doctest 4 months ago.

Today I'm happy to announce version 1.1 - which comes mainly with bugfixes and compile time improvements!

check out the /r/cpp thread for the discussion