gecko-dev/editor/docs/IMEHandlingGuide.rst

==================
IME handling guide
==================

This document explains how Gecko handles IME.

Introduction
============

IME is an abbreviation of Input Method Editor. This is a technical term from Windows but these days, this is used on other platforms as well.

IME is a helper application of a user's text input. It handles native key events before or after focused application (depending on the platform) and creates a composition string (a.k.a. preedit string), suggests a list of what the user attempts to input, commits composition string as a selected item off the list and commits composition string without any conversion. IME is used by Chinese, Japanese, Korean and Taiwan users for inputting Chinese characters because the number of them is beyond thousands and cannot be input from the keyboard directly. However, especially on mobile devices nowadays, IME is also used for inputting Latin languages like autocomplete. Additionally, IME may be used for handwriting systems or speech input systems on some platforms.

If IME is available on focused elements, we call that state "enabled". If IME is not fully available(i.e., user cannot enable IME), we call this state "disabled".

If IME is enabled but users use direct input mode (e.g., for inputting Latin characters), we call it "IME is closed". Otherwise, we call it "IME is open". (FYI: "open" is also called "active" or "turned on". "closed" is also called "inactive" or "turned off")

So, this document is useful when you're try to fix a bug for text input in Gecko.


Composition string and clauses
==============================

Typical Japanese IME can input two or more words into a composition string. When a user converts from Hiragana characters to Chinese characters the composition string, Japanese IME separates the composition string into multiple clauses. For example, if a user types "watasinonamaehanakanodesu", it's converted to Hiragana characters, "わたしのなまえはなかのです", automatically (In the following screenshots, the composition string has a wavy underline and the only one clause is called "raw input clause").

.. image:: inputting_composition_string.png
   :alt: Screenshot of raw composition string which is inputting Roman character mode of MS-IME (Japanese)

.. image:: raw_composition_string.png
   :alt: Screenshot of raw composition string whose all characters are Hiragana character (MS-IME, Japanese)

When a user presses ``Convert`` key, Japanese IME separates the composition string as "わたしの" (my), "なまえは" (name is) and "なかのです" (Nakano). Then, converts each clause with Chinese characters: "私の", "名前は" and "中野です" (In the following screenshot each clause is underlined and not connected adjacently. These clauses are called "converted clause").

.. image:: converted_composition_string.png
   :alt: Screenshot of converted composition string (MS-IME, Japanese)

If one or more clauses were not converted as expected, the user can choose one of the clauses with Arrow keys and look for the expected result form the list in the drop down menu (In the following screenshot, the clause with the thicker underline is called "selected clause").

.. image:: candidatewindow.png
   :alt: Screenshot of candidate window of MS-IME (Japanese) which converts the selected clause

Basically, composition string and each clause style is rendered by Gecko. And the drop down menu is created by IME.

Each clause is represented with selection in the editor. From chrome script, you can check it with ``nsISelectionController``. In native code, you can access it with either ``nsISelectionController`` or ``mozilla::SelectionType`` (the latter is recommended because of type safer). And editor sets these IME selections from ``mozilla::TextRangeType`` which are sent by ``mozilla::WidgetCompositionEvent`` as ``mozilla::TextRangeArray``. Following table explains the mapping between them.

.. table:: Selection types of each clause of composition string or caret

   +------------------------------------------------------------+---------------------------------------+-------------------------+-------------------------+
   |                                                            |`nsISelectionController`_              |`mozilla::SelectionType`_|`mozilla::TextRangeType`_|
   +============================================================+=======================================+=========================+=========================+
   |Caret                                                       |``SELECTION_NORMAL``                   |``eNormal``              |``eCaret``               |
   +------------------------------------------------------------+---------------------------------------+-------------------------+-------------------------+
   |Raw text typed by the user                                  |``SELECTION_IME_RAW_INPUT``            |``eIMERawClause``        |``eRawClause``           |
   +------------------------------------------------------------+---------------------------------------+-------------------------+-------------------------+
   |Selected clause of raw text typed by the user               |``SELECTION_IME_SELECTEDRAWTEXT``      |``eIMESelectedRawClause``|``eSelectedRawClause``   |
   +------------------------------------------------------------+---------------------------------------+-------------------------+-------------------------+
   |Converted clause by IME                                     |``SELECTION_IME_CONVERTEDTEXT``        |``eIMEConvertedClause``  |``eConvertedClause``     |
   +------------------------------------------------------------+---------------------------------------+-------------------------+-------------------------+
   |Selected clause by the user or IME and also converted by IME|``SELECTION_IME_SELECTEDCONVERTEDTEXT``|``eIMESelectedClause``   |``eSelectedClause``      |
   +------------------------------------------------------------+---------------------------------------+-------------------------+-------------------------+

Note that typically, "Selected clause of raw text typed by the user" isn't used because when composition string is already separated to multiple clauses, that means that the composition string has already been converted by IME at least once.

.. _nsISelectionController: https://searchfox.org/mozilla-central/source/dom/base/nsISelectionController.idl
.. _mozilla::SelectionType: https://searchfox.org/mozilla-central/source/dom/base/nsISelectionController.idl
.. _mozilla::TextRangeType: https://searchfox.org/mozilla-central/source/widget/TextRange.h

Modules handling IME composition
================================

widget
------

Each widget handles native IME events and dispatches ``WidgetCompositionEvent`` with ``mozilla::widget::TextEventDispatcher`` to represent the behavior of IME in the focused editor.

This is the only module that depends on the users platform. See also `Native IME handlers`_ section for the detail of each platform's implementation.

.. note::

   Android widget still does not use ``TextEventDispatcher`` to dispatch ``WidgetCompositionEvents``, see `bug 1137567 <https://bugzilla.mozilla.org/show_bug.cgi?id=1137567>`__.

mozilla::widget::TextEventDispatcher
------------------------------------

This class is used by native IME handler(s) on each platform. This capsules the logic to dispatch ``WidgetCompositionEvent`` and ``WidgetKeyboardEvent`` for making the behavior on each platform exactly same. For example, if ``WidgetKeyboardEvent`` should be dispatched when there is a composition is managed by this class in XP level. First of use, native IME handlers get the rights to use ``TextEventDispatcher`` with a call of ``BeginNativeInputTransaction()``. Then, ``StartComposition()``, ``SetPendingComposition()``, ``FlushPendingComposition()``, ``CommitComposition()``, etc. are available if ``BeginNativeInputTransaction()`` return true. These methods automatically manage composition state and dispatch ``WidgetCompositionEvent`` properly.

This is also used by ``mozilla::TextInputProcessor`` which can emulates (or implements) IME with chrome script. So, native IME handlers using this class means that the dispatching part is also tested by automated tests.

mozilla::WidgetCompositionEvent
-------------------------------

Internally, ``WidgetCompositionEvent`` represents native IME behavior. Its message is one of following values:

eCompositionStart
^^^^^^^^^^^^^^^^^

This is dispatched at starting a composition. This represents a DOM ``compositionstart`` event. The mData value is a selected string at dispatching the DOM event and it's automatically set by ``TextComposition``.

eCompositionUpdate
^^^^^^^^^^^^^^^^^^

This is dispatched by ``TextComposition`` when an ``eCompoitionChange`` will change the composition string. This represents a DOM ``compositionupdate`` event.

eCompositionEnd
^^^^^^^^^^^^^^^

This is dispatched by ``TextComposition`` when an ``eCompositionCommitAsIs`` or ``eComposiitonCommit`` event is dispatched. This represents a DOM ``compositionend`` event.

eCompositionChange
^^^^^^^^^^^^^^^^^^

This is used internally only. This is dispatched at modifying a composition string, committing a composition, changing caret position and/or changing ranges of clauses. This represents a DOM text event which is not in any standards. ``mRanges`` should not be empty only with this message.

eCompositionCommitAsIs
^^^^^^^^^^^^^^^^^^^^^^

This is used internally only. This is dispatched when a composition is committed with the string. The ``mData`` value should be always be an empty string. This causes a DOM text event without clause information and a DOM ``compositionend`` event.

eCompositionCommit
^^^^^^^^^^^^^^^^^^

This is used internally only. This is dispatched when a composition is committed with specific string. The ``mData`` value is the commit string. This causes a DOM text event without clause information and a DOM ``compositionend`` event.

.. table:: Table of event messages

   +--------------------------+-------------------------------------------+-------------------------------+-----------------------+----------------------+
   |                          |meaning of mData                           |who sets ``mData``?            |``mRanges``            |representing DOM event|
   +==========================+===========================================+===============================+=======================+======================+
   |``eCompositionStart``     |selected string before starting composition|``TextComposition``            |``nullptr``            |``compositionstart``  |
   +--------------------------+-------------------------------------------+-------------------------------+-----------------------+----------------------+
   |``eCompositionUpdate``    |new composition string                     |``TextComposition``            |``nullptr``            |``compositionupdate`` |
   +--------------------------+-------------------------------------------+-------------------------------+-----------------------+----------------------+
   |``eCompositionEnd``       |commit string                              |``TextComposition``            |``nullptr``            |``compositionend``    |
   +--------------------------+-------------------------------------------+-------------------------------+-----------------------+----------------------+
   |``eCompositionChange``    |new composition string                     |widget (or ``TextComposition``)|must not be ``nullptr``|``text``              |
   +--------------------------+-------------------------------------------+-------------------------------+-----------------------+----------------------+
   |``eCompositionCommitAsIs``|N/A (must be empty)                        |nobody                         |``nullptr``            |None                  |
   +--------------------------+-------------------------------------------+-------------------------------+-----------------------+----------------------+
   |``eCompositionCommit``    |commit string                              |widget (or ``TextComposition``)|``nullptr``            |None                  |
   +--------------------------+-------------------------------------------+-------------------------------+-----------------------+----------------------+

PresShell
---------

``PresShell`` receives the widget events and decides an event target from focused document and element. Then, it sends the events and the event target to ``IMEStateManager``.

mozilla::IMEStateManager
------------------------

``IMEStateManager`` looks for a ``TextComposition`` instance whose native IME context is same as the widget' which dispatches the widget event. If there is no proper ``TextComposition`` instance, it creates the instance. And it sends the event to the ``TextComposition`` instance.

Note that all instances of ``TextComposition`` are managed by ``IMEStateManager``. When an instance is created, it's registered to the list. When composition completely ends, it's unregistered from the list (and released automatically).

mozilla::TextComposition
------------------------

``TextComposition`` manages a composition and dispatches DOM ``compositionupdate`` events.

When this receives an ``eCompositionChange``, ``eCompositionCommit`` or ``eCompositionCommitAsIs`` event, it dispatches the event to the stored node which was the event target of ``eCompositionStart`` event. Therefore, this class guarantees that all composition events for a composition are fired on same element.

When this receives ``eCompositionChange`` or ``eCompositionCommit``, this checks if new composition string (or committing string) is different from the last data stored by the ``TextComposition``. If the composition event is changing the composition string, the ``TextComposition`` instance dispatches ``WidgetCompositionEvent`` with ``eCompositionUpdate`` into the DOM tree directly and modifies the last data. The ``eCompositionUpdate`` event will cause a DOM ``compositionupdate`` event.

When this receives ``eCompositionCommitAsIs`` or ``eCompositionCommit``, this dispatches an ``eCompositionEnd`` event which will cause a DOM ``compositionend`` event after dispatching ``eCompositionUpdate`` event and/or ``eCompositionChange`` event if necessary.

One of the other important jobs of this is, when a focused editor handles a dispatched ``eCompositionChange`` event, this modifies the stored composition string and its clause information. The editor refers the stored information for creating or modifying a text node representing a composition string.

And before dispatching ``eComposition*`` events, this class removes ASCII control characters from dispatching composition event's data in the default settings. Although, this can be disabled with ``"dom.compositionevent.allow_control_characters"`` pref.

Finally, this class guarantees that requesting to commit or cancel current composition to IME is perefored synchronously. See `Forcibly committing composition`_ section for the detail.

editor/libeditor
----------------

`mozilla::EditorEventListener <https://searchfox.org/mozilla-central/source/editor/libeditor/EditorEventListener.cpp>`__ listens for trusted DOM ``compositionstart``, ``text`` and ``compositionend`` events and notifies `mozilla::EditorBase <https://searchfox.org/mozilla-central/source/editor/libeditor/EditorBase.cpp>`__ and `mozilla::TextEditor <https://searchfox.org/mozilla-central/source/editor/libeditor/TextEditor.cpp>`__ of the events.

When ``EditorBase`` receives an ``eCompositionStart`` (DOM ``"compositionstart"``) event, it looks for a proper ``TextComposition`` instance and stores it.

When ``TextEditor`` receives an ``eCompositionChange`` (DOM ``"text"``) event, it creates or modifies a text node which includes the composition string and `mozilla::CompositionTransaction <https://searchfox.org/mozilla-central/source/editor/libeditor/CompositionTransaction.cpp>`__ (it was called ``IMETextTxn``) sets IME selections for representing the clauses of the composition string.

When ``EditorBase`` receives an ``eCompositionEnd`` (DOM ``"compositionend"``) event, it releases the stored ``TextComposition`` instance.

nsTextFrame
-----------
``nsTextFrame`` paints IME selections.

mozilla::IMEContentObserver
---------------------------

``IMEContentObserver`` observes various changes of a focused editor. When an editor or a windowless plugin gets focus, an instance is created, starts to observe and notifies widget of IME getting focus. When the editor or windowless plugin loses focus, it notifies widget of IME losing focus, stops observing everything and is released.

This class observes selection changes (caret position changes), text changes of a focused editor and layout changes (by reflow or scroll) of everything in the document. It depends on the result of ``nsIWidget::GetIMEUpdatePreference()`` what is observed.

When this notifies something of widget and/or IME, it needs to be safe to run script because notifying something may cause dispatching one or more DOM events and/or new reflow. Therefore, ``IMEContentObserver`` only stores which notification should be sent to widget and/or IME. Then, ``mozilla::IMEContentObserver::IMENotificationSender`` tries to send the pending notifications when it might become safe to do that. Currently, it's tried:

* after a native event is dispatched from ``PresShell::HandleEventInternal()``
* at changing focus from a windowless plugin
* when new focused editor receives DOM "focus" event

.. note::

   The 3rd timing may not be safe actually, but it causes a lot of oranges of automated tests.

See also `Notifications to IME`_ section for the detail of sending notifications.

Currently, ``WidgetQueryContentEvent`` is handled via ``IMEContentObserver`` because if it has a cache of selection, it can set reply of ``eQuerySelectedText`` event only with the cache. That is much faster than using ``ContentEventHandler``.

e10s support
============

Even when a remote process has focus, native IME handler in chrome process does its job. So, there is process boundary between native IME handler and focused editor. Unfortunately, it's not allowed to use syncronous communication from chrome process to a remote process. This means that chrome process (and also native IME and our native IME handler) cannot query the focused editor contents directly. For fixing this issue, we have ``ContentCache`` classes around process boundary.

mozilla::ContentCache
---------------------
This is a base class of ``ContentCacheInChild`` and ``ContentCacheInParent`` and IPC-aware. This has common members of them including all cache data:

``mText``
    Whole text in focused editor. This may be too big but IME may request all text in the editor.

    If we can separate editor contents per paragraph, moving selection between paragraphs generates pseudo focus move, we can reduce this size and runtime cost of ``ContentEventHandler``. However, we've not had a plan to do that yet. Note that Microsoft Word uses this hack.

``mCompositionStart``
    Offset of composition string in ``mText``. When there is no composition, this is ``UINT32_MAX``.

``mSelection::mAnchor``, ``mSelection::mFocus``
    Offset of selection anchor and focus in ``mText``.

``mSelection::mWritingMode``
    Writing mode at selection start.

``mSelection::mAnchorCharRect``, ``mSelection::mFocusCharRect``
    Next character rect of ``mSelection::mAnchor`` and ``mSelection::mFocus``. If correspoinding offset is end of the editor contents, its rect should be caret rect.

    These rects shouldn't be empty rect.

``mSelection::mRect``
    Unifiied character rect in selection range. When the selection is collapsed, this should be caret rect.

``mFirstRect``
    First character rect of ``mText``. When ``mText`` is empty string, this should be caret rect.

``mCaret::mOffset``
    Always same as selection start offset even when selection isn't collappsed.

``mCaret::mRect``
    Caret rect at ``mCaret::mOffset``. If caret isn't actually exists, it's computed with a character rect at the offset.

``mTextRectArray::mStart``
    If there is composition, ``mStart`` is same as ``mCompositionStart``.
    Otherwise, ``UINT32_MAX``.

``mTextRectArray::mRects``
    Each character rects of composition string.

``mEditorRect``
    The rect of editor element.

mozilla::ContentCacheInChild
----------------------------

This exists only in remote processes. This is created as a member of `PuppetWidget <https://searchfox.org/mozilla-central/source/widget/PuppetWidget.cpp>`__. When ``PuppetWidget`` receives notifications to IME from ``IMEContentObserver`` in the remote process, it makes this class modify its cached content. Then, this class do that with ``WidgetQueryContentEvents``. Finally, ``PuppetWidget`` sends the notification and ``ContentCacheInParent`` instance as ``ContentCache`` to its parent process.

mozilla::ContentCacheInParent
-----------------------------

This exists as a member of ``TabParent``. When ``TabParent`` receives notification from corresponding remote process, it assigns ``ContentCacheInParent`` new ``ContentCache`` and post the notification to ``ContentCacheInParent``. If all sent ``WidgetCompositionEvents`` and ``WidgetSelectionEvents`` are already handled in the remote process, ``ContentCacheInParent`` sending the notifications to widget.

And also this handles ``WidgetQueryContentEvents`` with its cache. Supported event messages of them are:

* ``eQuerySelectedText`` (only with ``SelectionType::eNormal``)
* ``eQueryTextContent``
* ``eQueryTextRect``
* ``eQueryCaretRect``
* ``eQueryEditorRect``

Additionally, this does not support query content events with XP line breakers but this must not be any problem since native IME handlers query contents with native line breakers.

``ContentCacheInParent`` also manages sent ``WidgetCompositionEvents`` and ``WidgetSelectionEvents``. After these events are handled in the remote process, ``TabParent`` receives it with a call of ``RecvOnEventNeedingAckHandled()``. Then, it calls ``ContentCacheInParent::OnEventNeedingAckHandled()``. Finally, ``ContentCacheInParent`` flushes pending notifications.

How do mozilla::TextComposition and mozilla::IMEStateManager work in e10s mode?
-------------------------------------------------------------------------------
In remote process, they work as non-e10s mode. On the other hand, they work specially in parent process.

When ``IMEStateManager`` in parent process receives ``eCompositionStart``, it creates ``TextComposition`` instance normally. However, if the event target has remote contents, ``TextComposition::DispatchCompositionEvent()`` directly sends the event to the remote process instead of dispatching the event into the target DOM tree in the process.

That means that even in a parent process, anybody can retrieve ``TextComposition`` instance, but it just does nothing in parent process.

``IMEStateManager`` works more complicated because IMEStateManagers in each processe need to negotiate about owner ship of managing input context.

When a remote process gets focus, temporarily, ``IMEStateManager`` in parent process disables IME in the widget. After that, ``IMEStateManager`` in the remote process will set proper input context for the focused editor. At this time, ``IMEStateManager`` in the parent process does nothing. Therefore, ``IMEContentObserver`` is never created while a remote process has focus.

When a remote process loses focus, ``IMEStateManager`` in parent process notifies ``IMEStateManager`` in the remote process of "Stop IME state management". When ``IMEStateManager::StopIMEStateManagement()`` is called in the remote process by this, the ``IMEStateManager`` forgets all focus information (i.e., that indicates nobody has focus).

When ``IMEStateManager`` in parent process is notified of pseudo focus move from or to menubar while a remote process has focus, it notifies the remote process of "Menu keyboard listener installed". Then, ``TabChild`` calls ``IMEStateManager::OnInstalledMenuKeyboardListener()`` in the remote process.

Style of each clause
--------------------

The style of each IME selection is managed by `LookAndFeel <https://searchfox.org/mozilla-central/source/widget/LookAndFeel.h>`__ class per platform. Therefore, it can be overridden by prefs.

Background color, foreground color (text color) and underline color can be specified with following prefs. The values must be string of "#rrggbb" format.

* ``ui.IMERawInputBackground``
* ``ui.IMERawInputForeground``
* ``ui.IMERawInputUnderline``
* ``ui.IMESelectedRawTextBackground``
* ``ui.IMESelectedRawTextForeground``
* ``ui.IMESelectedRawTextUnderline``
* ``ui.IMEConvertedTextBackground``
* ``ui.IMEConvertedTextForeground``
* ``ui.IMEConvertedTextUnderline``
* ``ui.IMESelectedConvertedTextBackground``
* ``ui.IMESelectedConvertedTextForeground``
* ``ui.IMESelectedConvertedTextUnderline``

Underline style can be specified with the following prefs. The values are integer,  0: none, 1: dotted, 2: dashed, 3: solid, 4: double, 5: wavy (The values same as ``NS_STYLE_TEXT_DECORATION_STYLE_*`` defined in `nsStyleConsts.h <https://searchfox.org/mozilla-central/source/layout/style/nsStyleConsts.h>`__.

* ``ui.IMERawInputUnderlineStyle``
* ``ui.IMESelectedRawTextUnderlineStyle``
* ``ui.IMEConvertedTextUnderlineStyle``
* ``ui.IMESelectedConvertedTextUnderlineStyle``

Underline width can be specified with ``"ui.IMEUnderlineRelativeSize"`` pref. This affects all types of clauses. The value should be 100 or 200. 100 means normal width, 200 means double width.

On some platforms, IME may support its own style for each clause. Currently, this feature is supported in TSF mode of Windows and on Linux. The style information is stored in ``TextRangeStyle`` which is defined in `TextRange.h <https://searchfox.org/mozilla-central/source/widget/TextRange.h>`__. It's a member of ``TextRange``. ``TextRange`` is stored in ``mRanges`` of ``WidgetCompositionEvent`` only when its message is ``eCompositionChange``.
Lifetime of composition string

When native IME notifies Gecko of starting a composition, a widget dispatches ``WidgetCompositionEvent`` with ``eCompositionStart`` which will cause a DOM ``compositionstart`` event.

When native IME notifies Gecko of a composition string change, a caret position change and/or a change of length of clauses, a widget dispatches ``WidgetCompositionEvent`` with ``eCompositionChange`` event. It will cause a DOM ``compositionupdate`` event when composition string is changing. That is dispatched by ``TextComposition`` automatically. After that when the widget and ``PresShell`` of the focused editor have not been destroyed yet, the ``eCompositionChange`` will cause a DOM text event which is not in any web standards.

When native IME notifies Gecko of the ending of a composition, a widget dispatches ``WidgetCompositionEvent`` with ``eCompositionCommitAsIs`` or ``eCompositionCommit``. If the committing string is different from the last set of data (i.e., if the event message is ``eCompositionCommit``), ``TextComposition`` dispatches a DOM ``compositionupdate`` event. After that, when the widget and ``PresShell`` of the focused editor have not been destroyed yet, an ``eCompositionChange`` event dispatched by ``TextComposition``, that causes a DOM text event. Finally, if the widget and PresShell of the focused editor has not been destroyed yet too, TextComposition dispatches an eCompositionEnd event which will cause a DOM compositionend event.
Limitation of handling composition

Currently, ``EditorBase`` touches undo stack at receiving every ``WidgetCompositionEvent``. Therefore, ``EditorBase`` requests to commit composition when the following cases occur:

* The editor loses focus
* The caret is moved by mouse or Javascript
* Value of the editor is changed by Javascript
* Node of the editor is removed from DOM tree
* Somethings object is modified in an HTML editor, e.g., resizing an image
* Composition string is moved to a different position which is specified by native IME (e.g., only a part of composition is committed)

In the future, we should fix this limitation. If we make ``EditorBase`` not touch undo stack until composition is committed, some of the cases must be fixed.

Notifications to IME
====================

XP part of Gecko uses ``nsIWidget::NotifyIME()`` for notifying ``widget`` of something useful to handle IME. Note that some of them are notified only when ``nsIWidget::GetIMEUpdatePreference()`` returns flags which request the notifications.

``NOTIFY_IME_OF_TEXT_CHANGE``, ``NOTIFY_IME_OF_SELECTION_CHANGE``, ``NOTIFY_IME_OF_POSITION_CHANGE`` and ``NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED`` are always sent by following order:

1. ``NOTIFY_IME_OF_TEXT_CHANGE``
2. ``NOTIFY_IME_OF_SELECTION_CHANGE``
3. ``NOTIFY_IME_OF_POSITION_CHANGE``
4. ``NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED``

If sending one of above notifications causes higher priority notification, the sender should abort to send remaning notifications and restart from highet priority notification again.

Additionally, all notifications except ``NOTIFY_IME_OF_BLUR`` should be sent only when it's safe to run script since the notification may cause querying content and/or dispatching composition events.

NOTIFY_IME_OF_FOCUS
-------------------

When an editable editor gets focus and ``IMEContentObserver`` starts to observe it, this is sent to widget. This must be called after the previous ``IMEContentObserver`` notified widget of ``NOTIFY_IME_OF_BLUR``.

Note that even if there are pending notifications, they are canceled when ``NOTIFY_IME_OF_FOCUS`` is sent since querying content with following notifications immediately after getting focus does not make sense. The result is always same as the result of querying contents at receiving this notfication.

NOTIFY_IME_OF_BLUR
------------------

When an ``IMEContentObserver`` instance ends observing the focused editor, this is sent to ``widget`` synchronously because assumed that this notification causes neither query content events nor composition events.

If ``widget`` wants notifications even while all windows are deactive, ``IMEContentObserver`` doesn't end observing the focused editor. I.e., in this case, ``NOTIFY_IME_OF_FOCUS`` and ``NOTIFY_IME_OF_BLUR`` are not sent to ``widget`` when a window which has a composition is being activated or deactivated.

When ``widget`` wants notifications during deactive, ``widget`` includes ``NOTIFY_DURING_DEACTIVE`` to the result of ``nsIWidget::GetIMEUpdatePreference()``.

If this notification is tried to sent before sending ``NOTIFY_IME_OF_FOCUS``, all pending notifications and ``NOTIFY_IME_OF_BLUR`` itself are canceled.

NOTIFY_IME_OF_TEXT_CHANGE
-------------------------

When text of focused editor is changed, this is sent to ``widget`` with a range of the change. But this is sent only when result of ``nsIWidget::GetIMEUpdatePreference()`` includes ``NOTIFY_TEXT_CHANGE``.

If two or more text changes occurred after previous ``NOTIFY_IME_OF_TEXT_CHANGE`` or ``NOTIFY_IME_OF_FOCUS``, the ranges of all changes are merged. E.g., if first change is from ``1`` to ``5`` and second change is from ``5`` to ``10``, the notified range is from ``1`` to ``10``.

If all merged text changes were caused by composition, ``IMENotification::mTextChangeData::mCausedOnlyByComposition`` is set to true. This is useful if native IME handler wants to ignore all text changes which are expected by native IME.

If at least one text change of the merged text changes was caused by current composition, ``IMENotification::mTextChangeData::mIncludingChangesDuringComposition`` is set to true. This is useful if native IME handler wants to ignore delayed text change notifications.

If at least one text change of the merged text changes was caused when there was no composition, ``IMENotification::mTextChangeData::mIncludingChangesWithoutComposition`` is set to true.

NOTIFY_IME_OF_SELECTION_CHANGE
------------------------------

When selection (or caret position) is changed in focused editor, widget is notified of this.

If the last selection change was occurred by a composition event event handling, ``IMENotification::mSelectionChangeData::mCausedByComposition`` is set to true. This is useful if native IME handler wants to ignore the last selection change which is expected by native IME.

If the last selection change was occurred by an ``eSetSelection`` event, ``IMENotification::mSelectionChangeData::mCausedBySelectionEvent`` is set to true. This is useful if native IME handler wants to ignore the last selection change which was requested by native IME.

If the last selection is occurred during a composition, ``IMENotification::mSelectionChangeData::mOccurredDuringComposition`` is set to true. This is useful if native IME handler wants to ignore the last selection change which occurred by web application's ``compositionstart`` or ``compositionupdate`` event handler before inserting composition string.

NOTIFY_IME_OF_POSITION_CHANGE
-----------------------------

When reflow or scroll occurs in the document, this is sent to widget. But this is sent only when result of ``nsIWidget::GetIMEUpdatePreference()`` includes ``NOTIFY_POSITION_CHANGE``.

This might be useful to update a candidate window position or something.

NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED
---------------------------------------

After ``TextComposition`` handles ``eCompositionStart``, ``eCompositionChange``, ``eComposiitionCommit`` or ``eCompositionCommitAsIs``, this notification is sent to widget. This might be useful to update a candidate window position or something.

NOTIFY_IME_OF_MOUSE_BUTTON_EVENT
--------------------------------

When a ``mousedown`` event or a ``mouseup`` event is fired on a character in a focused editor, this is sent to widget. But this is sent only when result of ``nsIWidget::GetIMEUpdatePreference()`` includes ``NOTIFY_MOUSE_BUTTON_EVENT_ON_CHAR``. This is sent with various information. See ``IMENotification::mMouseButtonEventData`` in `IMEData.h <https://searchfox.org/mozilla-central/source/widget/IMEData.h>`__ for the detail.

If native IME supports mouse button event handling, ``widget`` should notify IME of mouse button events with this. If IME consumes an event, ``widget`` should return ``NS_SUCCESS_EVENT_CONSUMED`` from ``nsIWidget::NotifyIME()``. Then, ``EditorBase`` doesn't handle the mouse event.

Note that if a ``mousedown`` event or a ``mouseup`` event is consumed by a web application (before a focused editor handles it), this notification is not sent to ``widget``. This means that web applications can handle mouse button events before IME.

Requests to IME
===============

XP part of Gecko can request IME to commit or cancel composition. This must be requested via ``IMEStateManager::NotifyIME()``. Then, ``IMEStateManager`` looks for a proper ``TextComposition`` instance. If it's found, ``TextComposition::RequestToCommit()`` for calling ``nsIWidget::NotifyIME()`` and handles some extra jobs.

widget should call the proper native API if it's available. Even if commit or canceling composition does not occur synchronously, widget doesn't need to emulate it since ``TextComposition`` will emulate it automatically. In other words, widget should only request to commit or cancel composition to IME.

REQUEST_TO_COMMIT_COMPOSITION
-----------------------------

A request to commit current composition to IME. See also following "`Forcibly committing composition`_" section for additional information.

REQUEST_TO_CANCEL_COMPOSITION
-----------------------------

A request to cancel current composition to IME. In other words, a request to commit current composition with an empty string.

Forcibly committing composition
===============================

When ``TextComposition::RequestToCommit()`` calls ``nsIWidget::NotifyIME()``, it guarantees synchronous commit or canceling composition.

In order to put it into practice, we need to handle the following four scenarios:

The composition is committed with non-empty string synchronously
----------------------------------------------------------------

This is the most usual case. In this case, ``TextComposition`` handles ``WidgetCompositionEvent`` instances during a request normally. However, in a remote process in e10s mode, this case never occurs since requests to native IME is handled asynchronously.

The composition is not committed synchronously but later
--------------------------------------------------------

This is the only case in a remote process in e10s mode or occurs on Linux even in non-e10s mode if the native IME is iBus. The callers of ``NotifyIME(REQUEST_TOCOMMIT_COMPOSITION)`` may expect that composition string is committed immediately for their next job. For such a case, ``TextComposition::RequestToCommit()`` synthesizes DOM composition events and a DOM text event for emulating to commit composition synchronously. Additionally, ``TextComposition`` ignores committing events which are dispatched by widget when the widget receives native IME events.

In this case, using the last composition string as commit string.

However, if the last composition string is only an ideographic space (fullwidth space), the composition string may be a placeholder of some old Chinese IME on Windows.

.. image:: ChangJie.png
   :alt: aScreenshot of ChangJie (Traditional Chinese IME) which puts an ideographic space into composition string for placeholder

In this case, although, we should not commit the placeholder character because it's not a character which the user wanted to input but we commit it as is. The reason is, inputting an ideographic space causes a composition. Therefore, we cannot distinguish if committing composition is unexpected. If the user uses such old Chinese IME, ``"intl.ime.remove_placeholder_character_at_commit"`` pref may be useful but we don't support them anymore in default settings (except if somebody will find a good way to fix this issue).

The composition is committed synchronously but with empty string
----------------------------------------------------------------

This case may occur on Linux or with some IME on other platforms. If a web application implements autocomplete, committing with different strings especially an empty string it might cause confusion.

In this case, TextComposition overwrites the commit string of ``eCompositionChange`` event dispatched by widget. However, if the last composition string is only an ideographic space, it shouldn't be committed. See the previous case

Note that this case doesn't work as expected when composition is in a remote process in e10s mode.

The composition is not committed
--------------------------------

On Linux, there is no API to request commit or canceling composition forcibly. Instead, Gecko uses ``gtk_im_context_reset()`` API for this purpose because most IME cancel composition with it. But there are some IMEs which do nothing when Gecko calls it.

If this occurs, Gecko should restart composition with a DOM ``compositionstart`` event , a DOM ``compositionupdate`` event and a DOM ``text`` event at caret position.

.. note::

   This issue hasn't been supported yet.

IME state management
====================

IME is a text input system. It means that except when a user wants to input some text, IME shouldn't be avaliable. For example, pressing the space key to attempt scrolling a page may be consumed and prevented by IME. Additionally, password editors need to request special behavior with IME.

For solving this issue, Gecko sets the proper IME state at DOM focus change.

First, when a DOM node gets focus, nsFocusManager notifies ``IMEStateManager`` of the new focused node (calls ``IMEStateManager::OnChangeFocus()``). ``IMEStateManager`` asks desired IME state by calling ``nsIContent::GetDesiredIMEState()`` of the node. If the node owns ``nsEditor`` instance, it asks for the desired IME state from the editor and returns the result.

Next, ``IMEStateManager`` initializes ``InputContext`` (defined in `IMEData.h <https://searchfox.org/mozilla-central/source/widget/IMEData.h>`__) with the desired IME state and node information. Then, it calls ``nsIWidget::SetInputContext()`` with the ``InputContext``.

Finally, widget stores the InputContext and enables or disables IME if the platform has such an API.

InputContext
------------

InputContext is a struct. Its ``mIMEState``, ``mHTMLInputType``, ``mHTMLInputInputMode`` and ``mActionHint`` are set at ``nsIWidget::SetInputContext()`` called.

mIMEState
^^^^^^^^^
IME state has two abilities. One is enabled state:

ENABLED
"""""""

This means IME is fully available. E.g., when an editable element such as ``<input type="text">``, ``<textarea>`` or ``<foo contentediable>`` has focus.

DISABLED
""""""""

This means IME is not available. E.g., when a non-editable element has focus or no element has focus, the desired IME state is ``DISABLED``.

PASSWORD
""""""""

This means IME state should be the same as the state when a native password field has focus. This state is set only when ``<input type="password"> (ime-mode: auto;)``, ``<input type="text" style="ime-mode: disabled;">`` or ``<textarea style="ime-mode: disabled;">``.

PLUGIN
""""""

This is set only when a windowless plugin has focus.

.. note::

   Be careful, even if a password field has focus, ``mIMEState`` may be ``ENABLED``. When you need to check if a password field has focus for security reasons, you should use ``InputContext::IsPasswordEditor()``.

The other is IME open state:

DONT_CHANGE_OPEN_STATE
""""""""""""""""""""""

The open state of IME shouldn't be changed. I.e., Gecko should keep the last IME open state.

OPEN
""""
Open IME. This is specified only when ime-mode of the new focused element is ``active``.

CLOSE
"""""
Close IME. This is specified only when ime-mode of the new focused element is ``inactive``.

.. note::

   E.g., on Linux, applications cannot manage IME open state. On such platforms, this is ignored.

.. note::

   IME open state should be changed only when ``nsIWidget::SetInputContext()`` is called at DOM focus change because changing IME open state while an editor has focus makes users confused. The reason why ``nsIWidget::SetInputContext()`` is called is stored in ``InputContextAction::mCause``.

How does Gecko disable IME in IMM mode on Windows
"""""""""""""""""""""""""""""""""""""""""""""""""

Every window on Windows is associated an ``IMContext``. When Gecko disables IME, `mozilla::widget::IMEHandler <https://searchfox.org/mozilla-central/source/widget/windows/WinIMEHandler.cpp>`__::SetInputContext() disasociates the context from the window.

How does Gecko disable IME in TSF mode on Windows
"""""""""""""""""""""""""""""""""""""""""""""""""

`mozilla::widget::TSFTextStore <https://searchfox.org/mozilla-central/source/widget/windows/TSFTextStore.cpp>`__ sets focus to a dummy context which disables the keyboard.

How does Gecko disable IME on Mac
"""""""""""""""""""""""""""""""""

`mozilla::widget::TextInputHandler <https://searchfox.org/mozilla-central/source/widget/cocoa/TextInputHandler.mm>`__::HandleKeyDownEvent() doesn't call focused view's interpretKeyEvents. This prevents native key events to be passed to IME.

How does Gecko disable IME on GTK
"""""""""""""""""""""""""""""""""

`mozilla::widget::IMContextWrapper <https://searchfox.org/mozilla-central/source/widget/gtk/IMContextWrapper.cpp>`__ sets focus to a dummy context which doesn't have IME composition.

How does Gecko disable IME on Android
"""""""""""""""""""""""""""""""""""""

?

mHTMLInputType
^^^^^^^^^^^^^^

The value is a string representing the focused editor.

``"text"``, ``"password"``, ``"number"``, etc.
    When an ``<input>`` element gets focus, the value is the type of the input element.

``"textarea"``
    When a ``<textarea>`` element gets focus, the value is ``"textarea"``.

``""``
    When an HTML editor (an element whose ``contenteditable`` attribute is "true" or document whose designMode is "on") gets focus, the value is empty. And also, when the other elements get focus.

mHTMLInputInputMode
^^^^^^^^^^^^^^^^^^^

The value is ``inputmode`` attribute value of the focused editor. This is set only when ``"dom.forms.inputmode"`` pref is true.

mActionHint
^^^^^^^^^^^

The value is ``enterkeyhint`` attribute value of the focused editor when ``"dom.forms.enterkeyhint"`` pref is true. This is useful for deciding the caption for the submit button in virtual keyboard. E.g., the value could be ``"Go"``, ``"Next"`` or ``"Search"``.

Native IME handlers
===================

Following classes handles IME on each platform:

Windows
-------

`mozilla::widget::IMEHandler`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This class manages input mehtod context of each window and makes ``IMMHandler`` or ``TSFTextStore`` work with active IME and focused editor or windowless plugin. This class has only static members, i.e., never created its instance.

__ https://searchfox.org/mozilla-central/source/widget/windows/WinIMEHandler.cpp

`mozilla::widget::IMMHandler`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This class is used when TSF mode is disabled by pref (``"intl.tsf.enable"``), focused content is a windowless plugin or active IME is for IMM (i.e., not TIP for TSF).

This class handles ``WM_IME_*`` messages and uses ``Imm*()`` API. This is a singleton class since Gecko supports only on IM context in a process. Typically, a process creates windows with default IM context. Therefore, this design is enough (ideally, an instance should be created per IM context, though). The singleton instance is created when it becomes necessary. So, if user doesn't meet a windowless plugin nor use IME on it, this instance is never created.

__ https://searchfox.org/mozilla-central/source/widget/windows/IMMHandler.cpp

`mozilla::widget::TSFTextStore`__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This class handles IME events in TSF mode and when TIP (IME implemented with TSF) is active. This instances are created when an editable element gets focus and released when it loses focus.

``TSFTextStore`` implements some COM interfaces which is necessary to work with TIP. And similarly, there is a singleton class, ``TSFStaticSink``, to observe active TIP changes.

TSF is the most complicated IME API on all platforms, therefore, design of this class is also very complicated.

FIrst, TSF/TIP requests to lock the editor content for querying or modifying the content or selection. However, web standards don't have such mechanim. Therefore, when it's requested, ``TSFTextStore`` caches current content and selection with ``WidgetQueryContentEvent``. Then, it uses the cache to reply to query requests, and modifies the cache as they requested. At this time, ``TSFTextStore`` saves the requests of modification into the queue called ``PendingAction``. Finally, after unlocking the contents, it flushes the pending actions with dispatches ``WidgetCompositionEvents`` via ``TextEventDispatcher``.

Then, ``IMEContentObserver`` will notify some changes caused by the dispatched ``WidgetCompositionEvents`` (they are notified synchronously in chrome or non-e10s mode, but asynchronously from a remote process in e10s mode). At this time, ``TSFTextStore`` may receive notifications which indicates web application changes the content differently from cache in ``TSFTextStore``. However, ``TSFTextStore`` ignores such fact temporarily until the composition is finished completely. The reason is that, notifying unexpected text or selection changes to TSF and/or TIP during composition may behave them odd.

When a composition is committed and it receives ``NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED``, ``TSFTextStore`` clears the cache of contents and notifying TSF of merged text changes and the last selection change if they are not caused by composition. By this step, TSF and TIP may sync its internal cache with actual contents.

Note that if new composition is started before ``NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED`` notification, ``TSFTextStore`` handles the new composition with cached contents which may be different from actual contents. So, e.g., reconvertion around caret may not work as unexpectd in such case, but we don't have a good solution for this issue.

On the other hand, ``TSFTextStore`` cannot cache character rects since if there are a lot of characters, caching the rects require a lot of CPU cost (to compute each rect) and memory. Therefore, ``TSFTextStore`` will use insertion point relative query for them `bug 1286157 <https://bugzilla.mozilla.org/show_bug.cgi?id=1286157>`__. Then, it can retrieve expected character's rect even if the cache of ``TSFTextStore`` is different from the actual contents because TIP typically needs caret position's character rect (for a popup to indicate current input mode or next word suggestion list) or first character rect of the target clause of current composition (for a candidate list window of conversion).

__ https://searchfox.org/mozilla-central/source/widget/windows/TSFTextStore.cpp

Mac
---

Both IME and key events are handled in `TextInputHandler.mm <https://searchfox.org/mozilla-central/source/widget/cocoa/TextInputHandler.mm>`__.

``mozilla::widget::TextInputHandlerBase`` is the most base class. ``mozilla::widget::PluginTextInputHandler`` inherits ``TextInputHandlerBase`` and handles key events and IME events on focused plugin. ``mozilla::widget::IMEInputHandler`` inherits ``TextInputHandlerBase`` and handles IME related events. ``mozilla::widget::TextInputHandler`` inherits ``TextInputHandlerBase`` and implements ``NSTextInput`` protocol of Cocoa. The instance is created per `nsChildView <https://searchfox.org/mozilla-central/source/widget/cocoa/nsChildView.mm>`__ instance.

GTK
---

`mozilla::widget::IMContextWrapper <https://searchfox.org/mozilla-central/source/widget/gtk/IMContextWrapper.cpp>`__ handles IME. The instance is created per top level window.

Android
-------

`org.mozilla.geckoview.GeckoEditable <https://searchfox.org/mozilla-central/source/mobile/android/geckoview/src/main/java/org/mozilla/geckoview/GeckoEditable.java>`__ handles native IME events and `mozilla::widget::GeckoEditableSupport <https://searchfox.org/mozilla-central/source/widget/android/GeckoEditableSupport.cpp>`__ dispatches ``Widget*Event``.