Community:SummerOfCode08:TSF

My attempts at adding Text Services Framework support to the Mozilla platform as part of Google Summer of Code 08. (wish me luck!)

Status

Wow only 1 more week to go before the suggested pencils-down date. The implementation's been pretty solid through my testings and right now I'm concentrating on writing an automatic test for it. The test is a C++ unit test and is basically capable of testing every aspect of the TSF implementation from widget down to content. As of now most of the test is done and hopefully I can finish it by the start of next week. --Darchons 20:57, 1 August 2008 (UTC)

Implemented methods from what used to be nsIKBStateControl. Now TSF support will not get turned off because an IME function was called in one of these methods. Also added support for text decorations. Not full-fledged display attributes support, but works just like the existing IME code. --Darchons 21:20, 15 July 2008 (PDT)

This week is GSoC midterm and I'm pretty much on track to finish this project by the end of GSoC. The majority of things are working and the second half of GSoC will be focused on more extensive testing and implementation of several advanced features on top of the basic TSF implementation right now.

Several Good Things(tm) that are planned for the 2nd half:

• Display attributes
  • Display attributes are mostly used for IME's. Examples include underlines that make using IME easier. In Mozilla's IME implementation they are called text decorations.
  • Currently Mozilla's IME implementation uses a custom set of styles to use for text decorations. However, TSF is more advanced in that it lets the IME decide which text decorations are best to use.

• Input scopes
  • The input scope of a textbox is basically the "purpose" of the textbox. For example, for the Location Bar in Firefox, its input scope would be "URL", because the textbox is used to contain URL's. If a textbox is numbers-only, its input scope would be "Numbers" because it's used for numbers. For an HTML file input, the input scope would be "File pathname", and so on.
  • Input scopes are used by some IME's and text services, most notably the Tablet Input Panel for more accurate response. If the current input scope is numbers, a circle written by the user would be interpreted as zero rather than the letter O.
  • If we have this in Mozilla, it will be very useful for a lot of tablet and speech recognition users.

Speaking of speech recognition, I have been goofing around with testing Windows Speech Recognition and it's working pretty well. It used to be that you cannot use WSR to dictate into textboxes in Firefox but with TSF support it's a breeze. I wrote a letter in Gmail by talking only and it was fun :) --Darchons 10:29, 8 July 2008 (PDT)

Now that the foundation work is there, new cool stuff will start to pop up :) The Tablet Input Panel is working now and the icon is showing up:

This shows that TSF is activated and also Mozilla textboxes (the address bar in this case) are successfully exposed to the system (i.e. the system has access to the contents of these textboxes). --Darchons 14:24, 1 July 2008 (PDT)

The end of week 5, and a majority of the text store (the essential part of a TSF implementation and therefore the project) is implemented, and for the unimplemented methods I have a pretty good idea of how to implement them.

• Mozilla is TSF-aware
• Traditional IME's work (but no visual composition indicator, i.e. underline)

I think I'm ready for my first patch and in terms of completeness my implementation is on par with that of the last patch submitted for the bug a few years ago. In terms of actual functionality my implementation is a lot better because my implementation exposes the actual text boxes in Gecko to TSF while the old patch which did not. --Darchons 08:17, 28 June 2008 (PDT)

This is the end of week 4, (end of "full-time" week 1 for me since I had school and exams the first three weeks :/), and I have made stubs for the implementation, implemented several easy-to-implement methods, and made a basic outline of my design. --Darchons 10:48, 20 June 2008 (PDT)

Outline/Progress

Development

• Startup code
  • nsWindow::nsWindow

• Shutdown code
  • nsWindow::~nsWindow

• Focus/blur code
  • ITextStateObserver::OnFocusChange

• ITextStoreACP implementation (24 out of 26 implemented)
  • AdviseSink
  • UnadviseSink
  • RequestLock
    • addnsWindow input block (not required now but would be nice to have)
  • GetStatus
  • QueryInsert
  • GetSelection
    • add NS_QUERY_SELECTED_RANGE_TEXT event
  • SetSelection
    • add NS_SELECTION_SET event
    • add NS_SELECTION_ADD event
  • GetText
    • (use NS_QUERY_TEXT_CONTENT event)
  • SetText
    • (call SetSelection then InsertTextAtSelection)
    • can we prevent painting during the method?
  • GetFormattedText
    • somehow make use of nsDataObj
  • GetEmbedded
  • QueryInsertEmbedded
  • InsertEmbedded
  • RequestSupportedAttrs
  • RequestAttrsAtPosition
  • RequestAttrsTransitioningAtPosition
  • FindNextAttrTransition
  • RetrieveRequestedAttrs
  • GetEndACP
  • GetActiveView
  • GetACPFromPoint
    • add hit test event
  • GetTextExt
    • NS_QUERY_TEXT_RECT
  • GetScreenExt
    • NS_QUERY_FRAME_RECT
  • GetWnd
  • InsertTextAtSelection
    • NS_TEXT_TEXT
  • InsertEmbeddedAtSelection

• Notification code (7 of 7 implemented)
  • OnAttrsChange (not supported)
  • OnStartEditTransaction (not supported)
  • OnEndEditTransaction (not supported)
  • OnLayoutChange (not used)
  • OnSelectionChange
    • nsIWidget::OnIMESelectionChange
  • OnStatusChange (not supported)
  • OnTextChange
    • nsIWidget::OnIMETextChange

• Display attributes support

• Input scope support
  • -moz-input-scope property

Testing/Debugging

• Writing tests
• Tablet input
• Speech recognition

Thoughts

I was an extension developer for a while before this, but it definitely feels different to be working directly on the Mozilla trunk :) and as with a lots of things there are some goods and bads

One major goodness is how easy it is to do some of the stuff that I was doing.

Extreme example: Getting a window handle (HWND) to the Firefox window. In my extension I had to, (there are probably easier ways to do this that I haven't found)

• Start off with a nsIDOMNode
• Check for accessible attribute on the node
• If exists, get the nsIAccessible
  • QI to nsIAccessNode
• If doesn't exist, get the accessibility service ("@mozilla.org/accessibilityService;1")
  • QI for nsIAccessibleRetrieval
  • Call nsIAccessibleRetrieval::getAccessibleFor to get a nsIAccessible
  • QI to nsIAccessNode
• Call nsIAccessNode::GetAccessibleDocument to get a nsIAccessibleDocument
• Call GetAccessibleDocument::GetWindowHandle to get the HWND

It's not very pretty and you have to hope that accessibility is working (and included in the build, which thankfully is the default)

Now, for what I'm doing now,

• Start off with a pointer to nsWindow
• Call nsWindow::GetWindowHandle to get the HWND

The difference between the two is pretty big.

Now one bad thing is that in my extension I could pretty much use everything that the platform offers, as long as it accomplishes my goals.

But right now I'm working under the widget component which is lower-level to content, layout, etc. And I cannot directly use most of the stuff from these high-level components from widget otherwise that would break the abstraction. So it feels more constricted in a way than when writing an extension. But there are alternatives and these shouldn't cause problems.

Quirks

Firefox 3 is crashing on me constantly, and most of them aren't even getting caught by the crash report thingy and they are getting caught be the crash report thingy now (thx Neatnate) but FF3 is still crashing a lot :(

Well turns out this is what's bugging me the whole time (no pun intended) bug 432467 (thanks Ted!)

Installing FF3.0.1 build 1 seems to have fixed the problem :)

Praises

I'm using my laptop, which has a 12.1" screen, for this project, and it definitely makes it a lot easier to code when I hook it up to the main desktop's 22" LCD.