TPESystem/Media/VideoThumbnail

This page is intended to have an overall summary of discussions in Bug#942078[1] and webapi mailing list[2].

Background

Currently video thumbnail generation is not a good method and there are some room to improve as described below.

1. Seek to 1/10th of the duration
   1. 1/10th may not be a good choice
      A better way is to pick the biggest frame of the first 10 key frames as Android does. That frame should contain much information.
   2. Sometimes seeking takes time
      Current seeking is a precise way which means it decodes from the closest key-frame to the requested seek frame. If the frame at 1/10 duration is far from key frame, it takes longer.
   3. Seek also applies to audio
      Video thumbnail has nothing to do with audio. Need to find a way to avoid audio process.
2. Use Canvas to draw image and save it as a jpeg file
   There is a full-size memory allocation used in Canvas to hold a decoded frame.

Possible Solutions

• Use existing APIs or move some implementations to JS
  For item#1.1~1.2, they could be solved by using a JavaScript module to parse video file and find the time of proper key-frame to seek it. This is under discussion in Bug#971645[3].
  For item#1.3, there seems no existing way to inform gecko not init audio part. Using muted is not a good idea, since in order to let feel latency from muted to unmuted, normally muted is just to make volume zero not uninit/disable audio part.
  For item#2, createImageBitmap()<http://www.whatwg.org/specs/web-apps/current-work/multipage/timers.html#dom-createimagebitmap> should be able to reduce the memory usage.But it needs to work with seeking which has unnecessary audio data handle.
  
• Introduce a new API, mozCreateThumbnail(double thumbnailTime, unsigned long width, unsigned long height)
  This is redefined from the original proposed API, mozGetThumbnail(double time);
  For item#1.1~1.3, with some modifications in MediaDecoderReader and MediaDecoderStateMachine in gecko could fix those problems. Application can also use the javascript parser module to find a proper time and set it via "thumbnailTime" parameter, or let gecko find a proper frame.
  For item#2, memory usage could be reduced in gecko implementations by setting "width" and "height".
  
  Example:
  var video = document.createElement('video');
  video.mozCreateThumbnail(time, width, height); //new
  video.src = url;
  video.ongetthumbnail = function() { //new
   :
  captureFrame(video, metadata,callback);
   :
  }