Media/WebRTC/Testing

We currently test WebRTC with two sets of tests:

• Standalone C++ unit and system tests.
• Full system mochitests.

C++ Unit Tests

The WebRTC code has extensive C++ unit and system tests based on the Google Test (https://code.google.com/p/googletest/) framework. These tests live in two locations:

• media/mtransport — the mtransport generic transport subsystem and associated utilities
• media/webrtc/signaling/test — the signaling system and the media handling system

As usual, you can run them individually by doing:

  make -C <directory>

Alternately, you can run a single C++ unit test by name, e.g.,

  media/mtransport/test/ice_unittest

Tinderbox runs all the tests in a given directory but for a variety of reasons some of the tests aren't run on Tinderbox. This is controlled by requiring certain environment variables to be set.

A number of tests have been disabled on Tinderbox because they either take a long time or have excessive oranges. To enable these, set:

   MOZ_WEBRTC_TESTS=1

Additionally, the TURN tests require you to specify a TURN server. However, we don't provide one for Tinderbox because we don't want to operate a public server and releng hasn't set one up in the test environment yet ( https://bugzilla.mozilla.org/show_bug.cgi?id=865296)

If you have a TURN server, you can point the tests at it by setting:

   TURN_SERVER_ADDRESS
   TURN_SERVER_USER
   TURN_SERVER_PASSWORD

Currently we have the following test programs, each of which consists of one or more individual test cases. Note that you can enumerate the test cases like so:

    obj-x86_64-apple-darwin10.8.0/media/mtransport/test/ice_unittest --gtest_list_tests

Individual tests can be run with --gtest_filter, like so:

    obj-x86_64-apple-darwin10.8.0/media/mtransport/test/ice_unittest '--gtest_gilter=*Turn*'

media/mtransport/test

• sockettransportservice_unittest — tests various uses of the Socket Transport Service
• nrappkit_unittest — tests Mozilla implementations of the nrappkit primitives (needed for ICE)
• runnable_utils_unittest — tests the runnable utils in runnable_utils.h
• turn_unittest — unit tests for TURN (do not run on Tinderbox)
• ice_unittest — overall tests for ICE
• sctp_unittest — tests for the SCTP code running on top of transport
• transport_unittests — system tests for the mtransport subsystem including the DTLS wrapper and ICE wrappers

media/webrtc/signaling/test

• mediaconduit_unittests — unit tests for the audio/video conduit wrappers around GIPS
• mediapipeline_unittest — unit tests for the media pipelines that plumb between the Media Stream Graph, GIPS, and RTP
• signaling_unittests — full system tests for the PeerConnection API.

Note that the media pipeline and signaling_unittests do not test video, due to the linkage reasons mentioned below.

Technical Notes on use of GTest

WebRTC's use of gtest is different from the conventional Firefox usage of gtest (the WebRTC tests were written first).

• We need to start XPCOM. This is done using code from mtransport_test_utils.h. You

invoke it as:

    int main(int argc, char **argv) {
   test_utils = new MtransportTestUtils();
 
   // Start the tests
   ::testing::InitGoogleTest(&argc, argv);
 
   int rv = RUN_ALL_TESTS();
   delete test_utils;
   return rv;
 }

• The thread derived from main() isn't used as the "main thread". We often spin up a separate

thread to act as the main thread. This lets us wait for events while that thread ostensibly runs.

• Due to Firefox's internal compilation, we have to separately compile a lot of files without MOZILLA_INTERNAL_API. This means that a number of

files are compiled twice, e.g., in mtransport/build and mtransport/standalone. We still had problems linking with some elements of Firefox and so, for instance, parts of the video system are not available from the unit tests.

Mochitests and Crashtests

We also have a series of Mochitests which live in dom/media/test/mochitest and dom/media/test/crashtest. These can be run via the usual Mochitest mechanisms.

These tests mostly cover:

• getUserMedia.
• End-to-end calling scenarios
• That the system responds correctly to various kinds of API misuse (e.g., wrong arguments, etc.)

Limitations/Future Work

The current tests have a number of limitations that we aim to fix at some point:

• All of the tests run on the same machine so we don't test NAT traversal.
• The tests depend on whatever the ambient network environment is.
• As noted above, a number of the tests don't run in Mozilla CI (though they do run on the Ad Hoc CI System)
• None of the tests currently use real audio/video devices (though see https://bugzilla.mozilla.org/show_bug.cgi?id=815002)

We eventually need to do something to test in more useful network environments. The current plan is:

• Modify the C++ unit tests so that they no longer use the real network but rather send packets to a simulated network which can emulate various kinds of NAT behavior.

• Stand up a real-world testbed with a variety of different NATs and run full-browser tests between them. Note that this requires having tests that run on >1 machine.

We also want to set up an "interop" testbed that will test Chrome talking to Firefox. This also requires a bunch of the same modifications to run in >1 browser