Networking/Archive/RTSP

Background

RTSP (Real-Time Streaming Protocol) provides initiation and VCR-like control of streaming presentations. RTSP was standardized in 1998. RTSP builds on existing standards: It closely resembles HTTP in operation, and it can use SDP for session description and RTP for media transport.
For more detail about RTSP, please refer to http://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol and RFC 2326.

Protocol Stack

The complete multimedia protocol stack is illustrated in this figure, which shows the relationship between the RTSP/RTP framework and the supporting setup and control protocols.

Call Setup and Control

Various call setup, control, and advertisement protocols can be used to start an RTP session, depending on the application scenario:

• For the purpose of starting interactive session, be it a voice telephony call or a video conference, there are two standards. The original standard in this area was ITU recommendation H.323, and more recently the IETF has defined the Session Initiation Protocol (SIP).
• For the purpose of starting a noninteractive session--for example, video on demand--the main standard is the RTSP.
• The original use of RTP was with IP multicast and the lightweight sessions model of conferencing. This design used the Session Announcement Protocol (SAP) and IP multicast to announce ongoing sessions, such as seminars and TV broadcasts, that were open to the public.

Session Description

Session Description Protocol (SDP) is a means of describing the session, which conveys information such as transport addresses on the media flows, the format of the media, the RTP payload formats and profile that are to be used, the times when the session is active, and the purpose of the session.
SDP is in text file format, so it is human-readable and can be easily parsed.

Terminology and Reference

• RTSP: Real Time Streaming Protocol. An application-level protocol for control over the delivery of data with real-time properties. Defined in RFC 2326.
• RTP: Real-time Transport Protocol. Defines a standardized packet format for delivering audio and video over IP networks. Defined in RFC 3550.
• RTCP: RTP Control Protocol. Provides out-of-band statistics and control information for an RTP session. Defined in RFC 3550.
• SDP: Session Description Protocol. A format for describing streaming media initialization parameters. Defined in RFC 4566.
• RTP audio video profile - http://en.wikipedia.org/wiki/RTP_audio_video_profile

History (Important Bugs)

• bug 831645 - Support rtsp streaming framework
  • This is the first bug of RTSP streaming feature on B2G, which implemented most of the RTSP framework in Firefox OS.

• bug 992568 - [RTSP] Refactor RtspChannel to support HTTP->RTSP redirection and rendering inside the browser
  • Although the basic framework was ready, before FFOS v1.4, RTSP on B2G is rendered in the video app and supports only <a> and <video> tags. We had made a decision (in https://bugzilla.mozilla.org/show_bug.cgi?id=963565#c13) to render RTSP directly inside the B2G browser app. This bug serves as the collective implementation of several features, including:
    • Support RTSP in URL bar and <iframe>.
    • Support HTTP->RTSP redirection.
    • Render RTSP inside the browser.

• bug 957937 - (b2g-RTSP-2.0) [META] RTSP for 2.0
  
  • This is the meta bug of RTSP features in v2.0. We made a huge improvement in quality and reliability of RTSP feature in this version.
    

Supported RTP Payload Formats (Codecs)

RTP payload formats define how particular media types are transported within RTP. Payload formats are referenced by RTP profiles, and they may also define certain properties of the RTP data transfer protocol.
For a complete list of RTP payload formats, refer to http://en.wikipedia.org/wiki/RTP_audio_video_profile.

This tables lists the payload formats (codecs) supported by our RTSP client right now.

Payload Type (PT)	Name	Type	Description	Comment
dynamic	H264	video	H.264 video (MPEG-4 Part 10)
dynamic	MP4A-LATM	audio	MPEG-4 Audio
dynamic	H263-1998	video	H.263 video, second version (1998)
dynamic	H263-2000	video	H.263 video, third version (2000)
dynamic (or profile)	AMR	audio	Adaptive Multi-Rate audio
dynamic (or profile)	AMR-WB	audio	Adaptive Multi-Rate Wideband audio (ITU-T G.722.2)
dynamic (or profile)	MP4V-ES	video	MPEG-4 Visual
dynamic (or profile)	mpeg4-generic	audio/video	MPEG-4 Elementary Streams

For unsupported payload formats, such as MP3, an error message will be displayed on the browser.

Implementation

Framework Overview

This class diagram illustrates the most important components in our current RTSP framework. We can divide the framework into four major sub-modules:

• HTML Media Element
• RTSP Channel
• RTSP Controller
• Stagefright Library

HTML Media Element

Code path:

m-c/content/html/content/public/HTMLMediaElement.h
m-c/content/html/content/src/HTMLMediaElement.cpp
m-c/content/media/MediaDecoder.h/cpp
m-c/content/media/MediaResource.h/cpp
m-c/content/media/RtspMediaResource.h/cpp

RtspMediaResource provides an interface to deliver and control RTSP media data to RtspOmxDecoder.
Note that the design and behavior of RTSP streaming are quite different from HTTP streaming. Please refer to the comments in RtspMediaResource.h for more detail.

RTSP Channel

Code path:

m-c/netwerk/protocol/RtspHandler.h/cpp
m-c/netwerk/protocol/RtspChannelChild.h/cpp
m-c/netwerk/protocol/RtspChannelParent.h/cpp

Unlike other channels in Necko, RtspChannelChild is a dummy channel used to aid MediaResource creation in HTMLMediaElement. Network control and data flows are managed by RtspController objects, which is also responsible for IPC with the parent process.
For now, the existence of RtspChannelParent is only to support HTTP->RTSP redirection.

RTSP Controller

Code path:

m-c/netwerk/base/public/nsIStreamingProtocolController.idl
m-c/netwerk/protocol/rtsp/controller/RtspController.h/cpp
m-c/netwerk/protocol/rtsp/controller/RtspControllerChild.h/cpp
m-c/netwerk/protocol/rtsp/controller/RtspControllerParent.h/cpp

Both RtspControllerChild and RtspController inherit from and implement the interfaces nsIStreamingProtocolListener and nsIStreamingProtocolController. As its name indicates, nsIStreamingProtocolController interface is used to control and retrieve meta data from a media stream.

Stagefright Library

Code path:

m-c/netwerk/protocol/rtsp/rtsp/RTSPSource.h/cpp
m-c/netwerk/protocol/rtsp/rtsp/RTSPConnectionHandler.h
m-c/netwerk/protocol/rtsp/rtsp/*

We reused the stagefright library from AOSP. Source files for RTSP client were mostly ported to the folder "netwerk/protocol/rtsp/rtsp".
However, please note that currently we still depend on the utilities from Android foundation, such as ABuffer, ALooper and AMessage, etc.
The class RTSPSource is used directly by RtspController to perform operations, such as establishing RTSP connection and sending RTSP requests.

Shutdown/Close RtspMediaResource

This sequence diagram illustrates a series of actions involved in closing RtspMediaResource.
This digram also reveals the components and their relationships between HTMLMediaElement and RTSP modules.

TCP-Interleaved RTP

• Bug: bug 996765 - [RTSP] Support TCP-interleaved RTP transport
• Reference: http://www.informit.com/articles/article.aspx?p=169578&seqNum=3

The following information is extracted from this reference.

In certain scenarios, the best-effort, dynamic port methods of UDP-based RTP, as described previously, are not suitable. Some environments might consider the allocation of dynamic source and destination UDP ports through firewalls to be something they can live happily without. Moreover, just the nature of the Layer 1 and Layer 2 transport mechanisms underlying the data delivery might not be suited to nonguaranteed UDP traffic. In either instance, RTSP allows for the negotiation of the RTP delivery of the media data to be interleaved into the existing TCP connection.

When interleaving, the client-to-server SETUP command has the following format:

C->S  SETUP rtsp://video.foocorp.com:554/streams/example.rm RTSP/1.0
      Cseq: 3
      Transport: rtp/avp/tcp; interleaved=0-1

The changeover in the preceding example is in the transport description. First, the transport mechanisms have changed to show that the RTP delivery must be over TCP rather than UDP. Second, the addition of the interleaved option shows that the RTP data should be interleaved and use channel identifiers 0 and 1—0 will be used for the RTP data and 1 will be used for the RTCP messages. To confirm the transport setup, the server will respond with confirmation and a Session ID as before:

S->C  RTSP/1.0 200 OK
      Cseq: 3
      Session: 12345678
      Transport: rtp/ avp/tcp; interleaved=0-1

The RTP and RTCP data can now be transmitted over the existing RTSP TCP connection with the server using the 0 and 1 identifiers to represent the relevant channel.

This activity diagram illustrates a series of actions to transit from DISCONNECTED to CONNECTING, CONNECTED and PLAYING states.
It also reveals the flow chart of failing back to TCP-interleaved RTP transport.

This diagram models the state machine of RTSPSource class.

Future Plan

The major features, improvements and issues of our RTSP streaming are listed below:

• New Features
  • Support live stream (bug 1054171 - Support live stream over RTSP/RTP)
  • Support slideshow video (bug 1048982 - Cannot play slideshow video over RTSP)
• Improvements
  • We are working on a serious frame drop issue right now (bug 1056187 - Frame drops observed during RTSP streaming).
  • Other performance issues are listed in bug 1063466.
  • Bug 992568 has several follow-up bugs. Trace them from bug 992568.
  • Reduce the odds of system crash caused from RTSP: bug 1045062 - Replace CHECK assertions by NS_ASSERTION or graceful assertions
• Automated Test
  • The meta bug is bug 998899 - [META][RTSP] Build automated tests for RTSP feature.
  • Right now we are working on bug 1032111 - Basic RTSP test case on ICS emulator.