Tuesday, May 29, 2012

Looking for a better JavaScript unit test tool


Let's consider the state of JavaScript unit tests and testing tools.



JsUnit



We already use it for some of our js code.



Pros



  • can be invoked from an ant build file

  • launches browser to run the tests

  • Eclipse plug-in



Cons



  • launches browser to run the tests

  • Does not support js file to write the unit test code: it has to be embedded inside an html file

  • it has not be updated for a few years



Note: There is a JsUnit (2).



RhinoUnit



Pros



  • ant driven

  • supports js file

  • very simple to use



Cons



  • Simulation of JavaScript engine: not advanced enough to support our code: I tried to run test code working with JsUnit: I encountered issue when loading our common JavaScript files



crosscheck



Pros



  • Can invoked from ant build file

  • Simulates real browser behaviour



Cons



  • Simulation of JavaScript engine from a limited number of browser versions

  • No activity for 2 years: it does not support Firefox versions 2.x nor 3.x



jsspec



Pros



  • Runs on actual browser



Cons



  • JavaScript only framework: cannot be called from ant build file



jspec



Pros



  • Runs on actual browser



Cons



  • Does not seem to support our code: I tried to run test code running with js unit: I encountered issue when loading our common JavaScript files

  • JavaScript only framework: cannot be called from ant build file,



Screw.unit



Pros



  • Runs on actual browser



Cons



  • JavaScript only framework: cannot be called from ant build file



Note: I did not try it but it is very similar to jsspec and jspec.



It looks like JsUnit is the only choice we have. Please note it is already used in other components. It is not perfect though because it does not provide an easy way to apply the TDD process for the following reasons: - It does not provide a simple and integrated way to run JavaScript unit test - It forces you to write the unit tests in a html file instead of a .js file, - It forces you to have a local installation of the JsUnit framework in order to avoid absolute hard coded path to reference js unit files.



As a consequence, you have to switch back and forth from you IDE and all the browsers we want to support while "TDDing" in JavaScript. It is feasible but I do not think it is very effective (I may be wrong here).



Also I still need to look at a proper JavaScript editor or Eclipse plug-in allowing easy referencing and refactoring. There is the JSTD one which is part of WTP, however I have not played with it enough in order to have a good idea of it.



As a conclusion, I do not think we have the tools to be fully "TDD" compliant. We can be TOD (Test Oriented Development), but I have not found any tool that allows us to work with JavaScript as efficiently as with Java. Once again, the ideal solution would be to have something identical to JUnit.



What JavaScript unit testing tools do you use?



Recent Entries:



Source: Tips4all

20 comments:

  1. TestSwarm

    John Resig (jQuery) has created a tool for distributed JavaScript testing, TestSwarm. Mainly for open source JavaScript projects, but TestSwarm is open source, so you can set up a server yourself for corporate testing. Although this might require that you to do some modifications.

    pros:


    Supports all major browsers/operating systems
    Run on multiple clients at once
    Don't need to run server/clients on development computer (no need for IE)
    Automatic run tests on all clients when you commit something (or whenever you modify the script to run the tests)
    Show history of test results pr commit
    Supports multiple JavaScript test frameworks
    Have test results for OS and browser versions
    Crowdsource to test in a multitude of browsers


    cons:


    Can not break your build through ant/maven
    Don't notice the test fail before commit
    No IDEplug-in


    http://ejohn.org/blog/javascript-testing-does-not-scale/

    TestSwarm architecture:



    JsTestDriver

    Some people at Google have also started on a distributed JavaScript tool, JsTestDriver. It is similar to TestSwarm, that it has a server, and clients connected. But it also has support for running tests from command line and plugins for Eclipse and IntelliJ!

    pros:


    Supports all major browsers/operating systems
    Run on multiple clients at once
    Don't need to run server/clients on development computer (no need for IE)
    Run tests from command line (jar) (can be integrated in ant/maven)
    Eclipse plugin
    IntelliJ plugin
    Supports multiple JavaScript test frameworks


    cons:


    Doesn't show results for os or browser version. Only browser names. It does however print out the version in the test results.
    No history of previous test results


    Overview of how JsTestDriver works at runtime:


    Eclipse plugin screenshot:



    Short intro video:
    http://www.youtube.com/watch?v=V4wYrR6t5gE

    YUI Yeti

    Yahoo now has their own JavaScript test run server, Yeti. Built with node.js. It should be able to run your existing YUI-tests as they are, and works in multiple browsers. Since it runs from the command-line I guess it's mostly similar to JsTestDriver.



    Announcement from August 25th 2010: http://www.yuiblog.com/blog/2010/08/25/introducing-yeti-the-yui-easy-testing-interface/

    If you have experience with this test-runner, please contribute with more info :)

    Project home: http://yuilibrary.com/projects/yeti/

    Jasmine



    This is a test-runner that might interest developers familiar with Ruby or Ruby on Rails. The syntax is based on RSpec that's used for testing in Rails projects.


    Jasmine is a behavior-driven development framework for testing your JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM.


    If you have experience with this test-runner, please contribute with more info :)

    Project home: https://github.com/pivotal/jasmine/

    Sinon

    Another great tool is sinon.js by Christian Johansen, the author of Test-Driven JavaScript Development. Best described by himself:


    Standalone test spies, stubs and mocks
    for JavaScript. No dependencies, works
    with any unit testing framework.



    http://tddjs.com/

    ReplyDelete
  2. Take a look at the Dojo Object Harness (DOH) unit test framework which is pretty much framework independent harness for JavaScript unit testing and doesn't have any Dojo dependencies. There is a very good description of it at Unit testing Web 2.0 applications using the Dojo Objective Harness.

    If you want to automate the UI testing (a sore point of many developers) — check out doh.robot (temporary down. update: other link http://dojotoolkit.org/reference-guide/util/dohrobot.html ) and dijit.robotx (temporary down). The latter is designed for an acceptance testing.
    Update:

    Referenced articles explain how to use them, how to emulate a user interacting with your UI using mouse and/or keyboard, and how to record a testing session, so you can "play" it later automatically.

    ReplyDelete
  3. The JavaScript section of the Wikipedia entry, List of Unit Testing Frameworks, provides a list of available choices. It indicates whether they work client-side, server-side, or both.

    ReplyDelete
  4. Chutzpah - A JavaScript Test Runner

    I created an open source project called Chutzpah which is a test runner for JavaScript unit tests. Chutzpah enables you to run JavaScript unit tests from the command line and from inside of Visual Studio. It also supports running in the TeamCity continuous integration server.

    ReplyDelete
  5. I've tried YUITest and Qunit. I find Qunit much simpler and easy to use.

    ReplyDelete
  6. YUI has a testing framework as well. This video from Yahoo! Theater is a nice introduction, although there are a lot of basics about TDD up front.

    This framework is generic and can be run against any JavaScript or JS library.

    ReplyDelete
  7. google-js-test: new javascript testing framework recently released by Google:
    http://code.google.com/p/google-js-test/


    Can be used without browser
    Style and semantics that resemble Google Test for C++.
    A built-in mocking framework that requires minimal boilerplate code (e.g. no $tearDown or $verifyAll) with style and semantics based on the Google C++ Mocking Framework.
    No real DOM (testing of DOM interaction still possible)
    Currently no binaries for Windows

    ReplyDelete
  8. We are now using Qunit with Pavlov and JSTestDriver all together. This approach works well for us.

    QUnit: http://docs.jquery.com/QUnit

    Pavlov: http://www.elijahmanor.com/2010/06/bdd-style-qunit-testing-aspnet-mvcs.html
    https://github.com/mmonteleone/pavlov

    jsTestDriver: http://slmoloch.blogspot.com/2009/08/how-to-run-jstestdriver-with-visual_02.html
    http://code.google.com/p/js-test-driver/

    ReplyDelete
  9. BusterJS

    There is also BusterJS from Christian Johansen, the author of Test Driven Javascript Development and the Sinon framework. From the site:


    Buster.JS is a new JavaScrpt testing framework. It does browser testing by automating test runs in actual browsers (think JsTestDriver), as well as Node.js testing.

    ReplyDelete
  10. You have "runs on actual browser" as a pro, but in my experience that is a con because it is slow. But what makes it valuable is the lack of sufficient JS emulation from the non-browser alternatives. It could be that if your JS is complex enough that only an in browser test will suffice, but there are a couple more options to consider:

    HtmlUnit: "It has fairly good JavaScript support (which is constantly improving) and is able to work even with quite complex AJAX libraries, simulating either Firefox or Internet Explorer depending on the configuration you want to use." If its emulation is good enough for your use then it will be much faster than driving a browser.

    But maybe HtmlUnit has good enough JS support but you don't like Java? Then maybe:

    Celerity: Watir API running on JRuby backed by HtmlUnit.

    or similarly

    Schnell: another JRuby wrapper of HtmlUnit.

    Of course if HtmlUnit isn't good enough and you have to drive a browser then you might consider Watir to drive your JS.

    ReplyDelete
  11. Check out the recent screencast covering recent changes to JSpec, you will like what you see ;) very sexy, powerful, extendable test framework.

    http://vision-media.ca/resources/javascript/jspec-032-javascript-testing-framework

    ReplyDelete
  12. You should have a look at env.js. See my blog for an example how to write unit tests with env.js.

    ReplyDelete
  13. I have tried JsTestDriver and it looks pretty good but unfortunately does not work with prototype framework. It is already reported issue http://code.google.com/p/js-test-driver/issues/detail?id=32&q=prototype

    ReplyDelete
  14. We added JUnit integration to our Java to Javascript code generator ST-JS (http://st-js.sourceforge.net). The framework generates to corresponding Javascript for both the tested code and the unit tests and sends the code to different browsers.

    There is no need for a separate server as the unit test runner opens the needed http port (and closes it once the tests finished). The framework manipulates the Java stacktrace so that the failed asserts are correctly displayed by the JUnit Eclipse plugin. Here is a simple example with jQuery and Mockjax:

    @RunWith(STJSTestDriverRunner.class)
    @HTMLFixture("<div id='fortune'></div>")

    @Scripts({ "classpath://jquery.js",
    "classpath://jquery.mockjax.js", "classpath://json2.js" })
    public class MockjaxExampleTest {
    @Test
    public void myTest() {
    $.ajaxSetup($map("async", false));
    $.mockjax(new MockjaxOptions() {
    {
    url = "/restful/fortune";
    responseText = new Fortune() {
    {
    status = "success";
    fortune = "Are you a turtle?";
    }
    };
    }
    });

    $.getJSON("/restful/fortune", null, new Callback3<Fortune, String, JQueryXHR>() {
    @Override
    public void $invoke(Fortune response, String p2, JQueryXHR p3) {
    if (response.status.equals("success")) {
    $("#fortune").html("Your fortune is: " + response.fortune);
    } else {
    $("#fortune").html("Things do not look good, no fortune was told");
    }

    }
    });
    assertEquals("Your fortune is: Are you a turtle?", $("#fortune").html());
    }

    private static class Fortune {
    public String status;
    public String fortune;
    }
    }

    ReplyDelete
  15. Another name to throw into the ring would be Selenium, which (using Selenium RC) can be driven (for example) from Java, which allows for nice integration with JUnit/Eclipse/Ant.
    The actual JavaScript code to be tested runs in a real web browser (yes, that is slow), all of the major ones are being supported.

    ReplyDelete
  16. You might also be interested in the unit testing framework that is part of qooxdoo, an open source RIA framework similar to Dojo, ExtJS, etc. but with quite a comprehensive tool chain.

    Try the online version of the testrunner. Hint: hit the gray arrow at the top left (should be made more obvious). It's a "play" button that runs the selected tests.

    To find out more about the JS classes that let you define your unit tests, see the online API viewer.

    For automated UI testing (based on Selenium RC), check out the Simulator project.

    ReplyDelete
  17. A different approach, but crosses boundaries well is WATIN, WATIR, and WATIJ. Instead of testing javascript, they test the page its on, including specific interactions. Harnessed in the unit test framework (junit, nunit) the results can be run along with your other unit tests including from your build process.

    http://watin.sourceforge.net/

    ReplyDelete
  18. MochiKit has a testing framework called SimpleTest that seems to have caught on.
    Here's a blog post from the original author.

    ReplyDelete
  19. It's been mentioned (@Aaron Digulla) but you should take a look at envjs (http://www.envjs.com) It's basically a DOM written in Javascript that can be run in Rhino (and i believe SpiderMonkey) so you get to run all your tests on your CI server... and it's open source.

    http://ejohn.org/blog/bringing-the-browser-to-the-server/

    ReplyDelete
  20. I've liked using Jasmine because of the its simple, generic approach, BDD-style tests, ability to configure, and for Java environments, it integrates well with Maven (can fail build and generate JUnit reports) and Jenkins (can report test results and history).

    ReplyDelete