IME handling guide¶
This document explains how Gecko handles IME.
IME is an abbreviation of Input Method Editor. This is a technical term from Windows but these days, this is used on other platforms as well.
IME is a helper application of a user’s text input. It handles native key events before or after focused application (depending on the platform) and creates a composition string (a.k.a. preedit string), suggests a list of what the user attempts to input, commits composition string as a selected item off the list and commits composition string without any conversion. IME is used by Chinese, Japanese, Korean and Taiwan users for inputting Chinese characters because the number of them is beyond thousands and cannot be input from the keyboard directly. However, especially on mobile devices nowadays, IME is also used for inputting Latin languages like autocomplete. Additionally, IME may be used for handwriting systems or speech input systems on some platforms.
If IME is available on focused elements, we call that state “enabled”. If IME is not fully available(i.e., user cannot enable IME), we call this state “disabled”.
If IME is enabled but users use direct input mode (e.g., for inputting Latin characters), we call it “IME is closed”. Otherwise, we call it “IME is open”. (FYI: “open” is also called “active” or “turned on”. “closed” is also called “inactive” or “turned off”)
So, this document is useful when you’re try to fix a bug for text input in Gecko.
Composition string and clauses¶
Typical Japanese IME can input two or more words into a composition string. When a user converts from Hiragana characters to Chinese characters the composition string, Japanese IME separates the composition string into multiple clauses. For example, if a user types “watasinonamaehanakanodesu”, it’s converted to Hiragana characters, “わたしのなまえはなかのです”, automatically (In the following screenshots, the composition string has a wavy underline and the only one clause is called “raw input clause”).
When a user presses
Convert key, Japanese IME separates the composition
string as “わたしの” (my), “なまえは” (name is) and “なかのです” (Nakano). Then,
converts each clause with Chinese characters: “私の”, “名前は” and “中野です” (In
the following screenshot each clause is underlined and not connected
adjacently. These clauses are called “converted clause”).
If one or more clauses were not converted as expected, the user can choose one of the clauses with Arrow keys and look for the expected result form the list in the drop down menu (In the following screenshot, the clause with the thicker underline is called “selected clause”).
Basically, composition string and each clause style is rendered by Gecko. And the drop down menu is created by IME.
Each clause is represented with selection in the editor. From chrome script,
you can check it with
nsISelectionController. In native code, you can
access it with either
(the latter is recommended because of type safer). And editor sets these IME
mozilla::TextRangeType which are sent by
following table explains the mapping between them.
Raw text typed by the user
Selected clause of raw text typed by the user
Converted clause by IME
Selected clause by the user or IME and also converted by IME
Note that typically, “Selected clause of raw text typed by the user” isn’t used because when composition string is already separated to multiple clauses, that means that the composition string has already been converted by IME at least once.
Modules handling IME composition¶
Each widget handles native IME events and dispatches
mozilla::widget::TextEventDispatcher to represent the behavior of IME
in the focused editor.
This is the only module that depends on the users platform. See also Native IME handlers section for the detail of each platform’s implementation.
Android widget still does not use
TextEventDispatcher to dispatch
This class is used by native IME handler(s) on each platform. This capsules the
logic to dispatch
making the behavior on each platform exactly same. For example, if
WidgetKeyboardEvent should be dispatched when there is a composition is
managed by this class in XP level. First of use, native IME handlers get the
rights to use
TextEventDispatcher with a call of
CommitComposition(), etc. are available if
BeginNativeInputTransaction() return true. These methods automatically
manage composition state and dispatch
This is also used by
mozilla::TextInputProcessor which can emulates (or
implements) IME with chrome script. So, native IME handlers using this class
means that the dispatching part is also tested by automated tests.
WidgetCompositionEvent represents native IME behavior. Its
message is one of following values:
This is dispatched at starting a composition. This represents a DOM
compositionstart event. The mData value is a selected string at dispatching
the DOM event and it’s automatically set by
This is dispatched by
TextComposition when an
change the composition string. This represents a DOM
This is dispatched by
TextComposition when an
eCompositionCommit event is dispatched. This represents a DOM
This is used internally only. This is dispatched at modifying a composition
string, committing a composition, changing caret position and/or changing
ranges of clauses. This represents a DOM text event which is not in any
mRanges should not be empty only with this message.
This is used internally only. This is dispatched when a composition is
committed with the string. The
mData value should be always be an empty
string. This causes a DOM text event without clause information and a DOM
This is used internally only. This is dispatched when a composition is
committed with specific string. The
mData value is the commit string. This
causes a DOM text event without clause information and a DOM
meaning of mData
representing DOM event
selected string before starting composition
new composition string
new composition string
must not be
N/A (must be empty)
PresShell receives the widget events and decides an event target from
focused document and element. Then, it sends the events and the event target to
IMEStateManager looks for a
TextComposition instance whose native IME
context is same as the widget’ which dispatches the widget event. If there is
TextComposition instance, it creates the instance. And it sends
the event to the
Note that all instances of
TextComposition are managed by
IMEStateManager. When an instance is created, it’s registered to the list.
When composition completely ends, it’s unregistered from the list (and released
TextComposition manages a composition and dispatches DOM
When this receives an
eCompositionCommitAsIs event, it dispatches the event to the stored node
which was the event target of
eCompositionStart event. Therefore, this
class guarantees that all composition events for a composition are fired on
When this receives
checks if new composition string (or committing string) is different from the
last data stored by the
TextComposition. If the composition event is
changing the composition string, the
TextComposition instance dispatches
eCompositionUpdate into the DOM tree
directly and modifies the last data. The
eCompositionUpdate event will
cause a DOM
When this receives
eCompositionEnd event which will cause a DOM
compositionend event after dispatching
eCompositionUpdate event and/or
eCompositionChange event if necessary.
One of the other important jobs of this is, when a focused editor handles a
eCompositionChange event, this modifies the stored composition
string and its clause information. The editor refers the stored information for
creating or modifying a text node representing a composition string.
And before dispatching
eComposition* events, this class removes ASCII
control characters from dispatching composition event’s data in the default
settings. Although, this can be disabled with
Finally, this class guarantees that requesting to commit or cancel current composition to IME is perefored synchronously. See Forcibly committing composition section for the detail.
EditorBase receives an
"compositionstart") event, it looks for a proper
instance and stores it.
TextEditor receives an
it creates or modifies a text node which includes the composition string and
(it was called
IMETextTxn) sets IME selections for representing the clauses
of the composition string.
EditorBase receives an
event, it releases the stored
nsTextFrame paints IME selections.
IMEContentObserver observes various changes of a focused editor. When a
corresponding element of a
HTMLEditor instance gets
focus, an instance is created by
IMEStateManager, then, starts to observe
widget of IME getting focus. When the editor loses focus, it
widget of IME losing focus and stops observing everything.
Finally, it’s destroyed by
This class observes selection changes (caret position changes), text changes of
a focused editor and layout changes (by reflow or scroll) of everything in the
document. It depends on the result of
what is observed.
When this notifies
widget of something, it needs to be safe to run
script because notifying something may cause dispatching one or more DOM events
and/or new reflow. Therefore,
IMEContentObserver only stores which
notification should be sent to
mozilla::IMEContentObserver::IMENotificationSender tries to send the
pending notifications when it might become safe to do that. Currently, it’s
after a native event is dispatched from
when new focused editor receives DOM
when next refresh driver tick
The 3rd timing may not be safe actually, but it causes a lot of oranges of automated tests.
See also Notifications to IME section for the detail of sending notifications.
WidgetQueryContentEvent is handled via
because if it has a cache of selection, it can set reply of
eQuerySelectedText event only with the cache. That is much faster than
Even when a remote process has focus, native IME handler in chrome process does
its job. So, there is process boundary between native IME handler and focused
editor. Unfortunately, it’s not allowed to use synchronous communication from
chrome process to a remote process. This means that chrome process (and also
native IME and our native IME handler) cannot query the focused editor contents
directly. For fixing this issue, we have
ContentCache classes around
This is a base class of
and IPC-aware. This has common members of them including all cache data:
Whole text in focused editor. This may be too big but IME may request all text in the editor.
If we can separate editor contents per paragraph, moving selection between paragraphs generates pseudo focus move, we can reduce this size and runtime cost of
ContentEventHandler. However, we’ve not had a plan to do that yet. Note that Microsoft Word uses this hack.
Offset of composition string in
mText. When there is no composition, this is
Offset of selection anchor and focus in
Writing mode at selection start.
Next character rectangle of
mSelection::mFocus. If corresponding offset is end of the editor contents, its rectangle should be a caret rectangle.
These rectangles shouldn’t be empty rect.
Unified character rectangle in selection range. When the selection is collapsed, this should be caret rect.
First character rect of
mTextis empty string, this should be caret rect.
Always same as selection start offset even when selection isn’t collapsed.
Caret rect at
mCaret::mOffset. If caret isn’t actually exists, it’s computed with a character rect at the offset.
If there is composition,
mStartis same as
Each character rectangle of composition string.
The rect of editor element.
This exists only in remote processes. This is created as a member of
PuppetWidget receives notifications to IME from
in the remote process, it makes this class modify its cached content. Then,
this class do that with
sends the notification and
ContentCacheInParent instance as
ContentCache to its parent process.
This exists as a member of
notification from corresponding remote process, it assigns
ContentCache and post the notification to
ContentCacheInParent. If all sent
WidgetSelectionEvents are already handled in the remote process,
ContentCacheInParent sending the notifications to widget.
And also this handles
WidgetQueryContentEvents with its cache. Supported
event messages of them are:
Additionally, this does not support query content events with XP line breakers but this must not be any problem since native IME handlers query contents with native line breakers.
ContentCacheInParent also manages sent
WidgetSelectionEvents. After these events are handled in the remote
TabParent receives it with a call of
RecvOnEventNeedingAckHandled(). Then, it calls
ContentCacheInParent flushes pending notifications.
How do mozilla::TextComposition and mozilla::IMEStateManager work in e10s mode?¶
In remote process, they work as non-e10s mode. On the other hand, they work specially in parent process.
IMEStateManager in parent process receives
TextComposition instance normally. However, if the event target has
TextComposition::DispatchCompositionEvent() directly sends
the event to the remote process instead of dispatching the event into the
target DOM tree in the process.
That means that even in a parent process, anybody can retrieve
TextComposition instance, but it just does nothing in parent process.
IMEStateManager works more complicated because
IMEStateManager in each
process need to negotiate about owner ship of managing input context.
When a remote process gets focus, temporarily,
IMEStateManager in parent
process disables IME in the widget. After that,
IMEStateManager in the
remote process will set proper input context for the focused editor. At this
IMEStateManager in the parent process does nothing. Therefore,
IMEContentObserver is never created while a remote process has focus.
When a remote process loses focus,
IMEStateManager in parent process
IMEStateManager in the remote process of
“Stop IME state management”. When
is called in the remote process by this, the
IMEStateManager forgets all
focus information (i.e., that indicates nobody has focus).
IMEStateManager in parent process is notified of pseudo focus move
from or to menubar while a remote process has focus, it notifies the remote
process of “Menu keyboard listener installed”. Then,
IMEStateManager::OnInstalledMenuKeyboardListener() in the remote process.
Style of each clause¶
The style of each IME selection is managed by LookAndFeel class per platform. Therefore, it can be overridden by prefs.
Background color, foreground color (text color) and underline color can be specified with following prefs. The values must be string of “#rrggbb” format.
Underline style can be specified with the following prefs. The values are
integer, 0: none, 1: dotted, 2: dashed, 3: solid, 4: double, 5: wavy (The
values same as
mozilla::StyleTextDecorationStyle defined in
Underline width can be specified with
This affects all types of clauses. The value should be 100 or 200. 100 means
normal width, 200 means double width.
On some platforms, IME may support its own style for each clause. Currently,
this feature is supported in TSF mode of Windows and on Linux. The style
information is stored in
TextRangeStyle which is defined in
It’s a member of
TextRange is stored in
WidgetCompositionEvent only when its message is
Lifetime of composition string¶
When native IME notifies Gecko of starting a composition, a widget dispatches
eCompositionStart which will cause a DOM
When native IME notifies Gecko of a composition string change, a caret position
change and/or a change of length of clauses, a widget dispatches
eCompositionChange event. It will cause a
compositionupdate event when composition string is changing. That is
TextComposition automatically. After that when the widget and
PresShell of the focused editor have not been destroyed yet, the
eCompositionChange will cause a DOM text event which is not in any web
When native IME notifies Gecko of the ending of a composition, a widget
eCompositionCommit. If the committing string is different from the last set
of data (i.e., if the event message is
TextComposition dispatches a DOM
compositionupdate event. After that,
when the widget and
PresShell of the focused editor have not been destroyed
eCompositionChange event dispatched by
causes a DOM text event. Finally, if the widget and PresShell of the focused
editor has not been destroyed yet too,
TextComposition dispatches an
eCompositionEnd event which will cause a DOM compositionend event.
Limitation of handling composition¶
EditorBase touches undo stack at receiving every
EditorBase requests to commit
composition when the following cases occur:
The editor loses focus
Node of the editor is removed from DOM tree
Somethings object is modified in an HTML editor, e.g., resizing an image
Composition string is moved to a different position which is specified by native IME (e.g., only a part of composition is committed)
In the future, we should fix this limitation. If we make
touch undo stack until composition is committed, some of the cases must be
Notifications to IME¶
XP part of Gecko uses
nsIWidget::NotifyIME() for notifying
something useful to handle IME. Note that some of them are notified only when
nsIWidget::GetIMEUpdatePreference() returns flags which request the
NOTIFY_IME_OF_COMPOSITION_EVENT_HANDLED are always sent by following order:
If sending one of above notifications causes higher priority notification, the sender should abort to send remaining notifications and restart from high priority notification again.
Additionally, all notifications except
NOTIFY_IME_OF_BLUR should be sent
only when it’s safe to run script since the notification may cause querying
content and/or dispatching composition events.
When an editable editor gets focus and
IMEContentObserver starts to observe
it, this is sent to widget. This must be called after the previous
IMEContentObserver notified widget of
Note that even if there are pending notifications, they are canceled when
NOTIFY_IME_OF_FOCUS is sent since querying content with following
notifications immediately after getting focus does not make sense. The result
is always same as the result of querying contents at receiving this
IMEContentObserver instance ends observing the focused editor, this
is sent to
widget synchronously because assumed that this notification
causes neither query content events nor composition events.
widget wants notifications even while all windows are inactive,
IMEContentObserver doesn’t end observing the focused editor. I.e., in this
NOTIFY_IME_OF_BLUR are not sent to
widget when a window which has a composition is being activated or
widget wants notifications during inactive,
NOTIFY_DURING_DEACTIVE to the result of
If this notification is tried to sent before sending
all pending notifications and
NOTIFY_IME_OF_BLUR itself are canceled.
When text of focused editor is changed, this is sent to
widget with a range
of the change. But this is sent only when result of
If two or more text changes occurred after previous
NOTIFY_IME_OF_FOCUS, the ranges of all
changes are merged. E.g., if first change is from
5 and second
change is from
10, the notified range is from
If all merged text changes were caused by composition,
IMENotification::mTextChangeData::mCausedOnlyByComposition is set to true.
This is useful if native IME handler wants to ignore all text changes which are
expected by native IME.
If at least one text change of the merged text changes was caused by current
IMENotification::mTextChangeData::mIncludingChangesDuringComposition is set
to true. This is useful if native IME handler wants to ignore delayed text
If at least one text change of the merged text changes was caused when there
was no composition,
set to true.
When selection (or caret position) is changed in focused editor, widget is notified of this.
If the last selection change was occurred by a composition event event
set to true. This is useful if native IME handler wants to ignore the last
selection change which is expected by native IME.
If the last selection change was occurred by an
IMENotification::mSelectionChangeData::mCausedBySelectionEvent is set to
true. This is useful if native IME handler wants to ignore the last selection
change which was requested by native IME.
If the last selection is occurred during a composition,
IMENotification::mSelectionChangeData::mOccurredDuringComposition is set to
true. This is useful if native IME handler wants to ignore the last selection
change which occurred by web application’s
compositionupdate event handler before inserting composition string.
When reflow or scroll occurs in the document, this is sent to widget, but this
is sent only when result of
This might be useful to update a candidate window position or something.
this notification is sent to widget. This might be useful to update a candidate
window position or something.
Requests to IME¶
XP part of Gecko can request IME to commit or cancel composition. This must be
for a proper
TextComposition instance. If it’s found,
TextComposition::RequestToCommit() for calling
and handles some extra jobs.
widget should call the proper native API if it’s available. Even if commit or
canceling composition does not occur synchronously, widget doesn’t need to
emulate it since
TextComposition will emulate it automatically. In other
words, widget should only request to commit or cancel composition to IME.
A request to commit current composition to IME. See also following “Forcibly committing composition” section for additional information.
A request to cancel current composition to IME. In other words, a request to commit current composition with an empty string.
Forcibly committing composition¶
it guarantees synchronous commit or canceling composition.
In order to put it into practice, we need to handle the following four scenarios:
The composition is committed with non-empty string synchronously¶
This is the most usual case. In this case,
WidgetCompositionEvent instances during a request normally. However, in a
remote process in e10s mode, this case never occurs since requests to native
IME is handled asynchronously.
The composition is not committed synchronously but later¶
This is the only case in a remote process in e10s mode or occurs on Linux even
in non-e10s mode if the native IME is iBus. The callers of
NotifyIME(REQUEST_TO_COMMIT_COMPOSITION) may expect that composition string
is committed immediately for their next job. For such a case,
TextComposition::RequestToCommit() synthesizes DOM composition events and a
DOM text event for emulating to commit composition synchronously. Additionally,
TextComposition ignores committing events which are dispatched by widget
when the widget receives native IME events.
In this case, using the last composition string as commit string.
However, if the last composition string is only an ideographic space (fullwidth space), the composition string may be a placeholder of some old Chinese IME on Windows.
In this case, although, we should not commit the placeholder character because
it’s not a character which the user wanted to input but we commit it as is. The
reason is, inputting an ideographic space causes a composition. Therefore, we
cannot distinguish if committing composition is unexpected. If the user uses
such old Chinese IME,
pref may be useful but we don’t support them anymore in default settings
(except if somebody will find a good way to fix this issue).
The composition is committed synchronously but with empty string¶
This case may occur on Linux or with some IME on other platforms. If a web application implements autocomplete, committing with different strings especially an empty string it might cause confusion.
In this case, TextComposition overwrites the commit string of
eCompositionChange event dispatched by widget. However, if the last
composition string is only an ideographic space, it shouldn’t be committed. See
the previous case.
Note that this case doesn’t work as expected when composition is in a remote process in e10s mode.
The composition is not committed¶
On Linux, there is no API to request commit or canceling composition forcibly.
Instead, Gecko uses
gtk_im_context_reset() API for this purpose because
most IME cancel composition with it. But there are some IMEs which do nothing
when Gecko calls it.
If this occurs, Gecko should restart composition with a DOM
compositionstart event , a DOM
compositionupdate event and a DOM
text event at caret position.
This issue hasn’t been supported yet.
IME state management¶
IME is a text input system. It means that except when a user wants to input some text, IME shouldn’t be available. For example, pressing the space key to attempt scrolling a page may be consumed and prevented by IME. Additionally, password editors need to request special behavior with IME.
For solving this issue, Gecko sets the proper IME state at DOM focus change.
First, when a DOM node gets focus, nsFocusManager notifies
of the new focused node (calls
IMEStateManager asks desired IME state by calling
nsIContent::GetDesiredIMEState() of the node. If the node owns
TextEditor instance, it asks for the desired IME state from the editor and
returns the result.
InputContext (defined in
with the desired IME state and node information. Then, it calls
nsIWidget::SetInputContext() with the
Finally, widget stores the InputContext and enables or disables IME if the platform has such an API.
InputContext is a struct. Its
mActionHint are set at
IME state has two abilities. One is enabled state:
This means IME is fully available. E.g., when an editable element such as
<foo contenteditable> has focus.
This means IME is not available. E.g., when a non-editable element has focus or
no element has focus, the desired IME state is
This means IME state should be the same as the state when a native password
field has focus. This state is set only when
<input type="password"> (ime-mode: auto;),
<input type="text" style="ime-mode: disabled;"> or
<textarea style="ime-mode: disabled;">.
The other is IME open state:
The open state of IME shouldn’t be changed. I.e., Gecko should keep the last IME open state.
Open IME. This is specified only when ime-mode of the new focused element is
Close IME. This is specified only when ime-mode of the new focused element is
E.g., on Linux, applications cannot manage IME open state. On such platforms, this is ignored.
IME open state should be changed only when
is called at DOM focus change because changing IME open state while an
editor has focus makes users confused. The reason why
nsIWidget::SetInputContext() is called is stored in
How does Gecko disable IME in IMM mode on Windows¶
Every window on Windows is associated an
IMContext. When Gecko disables
disassociates the context from the window.
How does Gecko disable IME in TSF mode on Windows¶
mozilla::widget::TSFTextStore sets focus to a dummy context which disables the keyboard.
How does Gecko disable IME on Mac¶
mozilla::widget::TextInputHandler::HandleKeyDownEvent() doesn’t call focused view’s interpretKeyEvents. This prevents native key events to be passed to IME.
How does Gecko disable IME on GTK¶
mozilla::widget::IMContextWrapper sets focus to a dummy context which doesn’t have IME composition.
How does Gecko disable IME on Android¶
The value is a string representing the focused editor.
<input>element gets focus, the value is the type of the input element.
<textarea>element gets focus, the value is
When an HTML editor (an element whose
trueor document whose
"on") gets focus, the value is empty. And also, when the other elements get focus.
The value is
inputmode attribute value of the focused editor.
The value is
enterkeyhint attribute value of the focused editor when
"dom.forms.enterkeyhint" pref is true. This is useful for deciding the
caption for the submit button in virtual keyboard. E.g., the value could be
Native IME handlers¶
Following classes handles IME on each platform:
This class manages input method context of each window and makes
TSFTextStore work with active IME and focused editor. This class has
only static members, i.e., never created its instance.
This class is used when TSF mode is disabled by pref (
since 108, formerly named
"intl.tsf.enable") or active IME is for IMM
(i.e., not TIP for TSF).
This class handles
WM_IME_* messages and uses
Imm*() API. This is a
singleton class since Gecko supports only on IM context in a process.
Typically, a process creates windows with default IM context. Therefore, this
design is enough (ideally, an instance should be created per IM context,
though). The singleton instance is created when it becomes necessary.
This class handles IME events in TSF mode and when TIP (IME implemented with TSF) is active. This instances are created when an editable element gets focus and released when it loses focus.
TSFTextStore implements some COM interfaces which is necessary to work with
TIP. And similarly, there is a singleton class,
TSFStaticSink, to observe
active TIP changes.
TSF is the most complicated IME API on all platforms, therefore, design of this class is also very complicated.
FIrst, TSF/TIP requests to lock the editor content for querying or modifying
the content or selection. However, web standards don’t have such mechanism.
Therefore, when it’s requested,
TSFTextStore caches current content and
WidgetQueryContentEvent. Then, it uses the cache to reply to
query requests, and modifies the cache as they requested. At this time,
TSFTextStore saves the requests of modification into the queue called
PendingAction. Finally, after unlocking the contents, it flushes the
pending actions with dispatches
IMEContentObserver will notify some changes caused by the dispatched
WidgetCompositionEvents (they are notified synchronously in chrome or
non-e10s mode, but asynchronously from a remote process in e10s mode). At this
TSFTextStore may receive notifications which indicates web
application changes the content differently from cache in
TSFTextStore ignores such fact temporarily until the composition
is finished completely. The reason is that, notifying unexpected text or
selection changes to TSF and/or TIP during composition may behave them odd.
When a composition is committed and it receives
TSFTextStore clears the cache
of contents and notifying TSF of merged text changes and the last selection
change if they are not caused by composition. By this step, TSF and TIP may
sync its internal cache with actual contents.
Note that if new composition is started before
handles the a composition with cached contents which may be different from
actual contents. So, e.g., reconversion around caret may not work as unexpected
in such case, but we don’t have a good solution for this issue.
On the other hand,
TSFTextStore cannot cache character rectangles since if
there are a lot of characters, caching the rectangles require a lot of CPU cost
(to compute each rect) and memory. Therefore,
TSFTextStore will use
insertion point relative query for them
bug 1286157. Then,
it can retrieve expected character’s rect even if the cache of
is different from the actual contents because TIP typically needs caret
position’s character rect (for a popup to indicate current input mode or next
word suggestion list) or first character rect of the target clause of current
composition (for a candidate list window of conversion).
Both IME and key events are handled in TextInputHandler.mm.
mozilla::widget::TextInputHandlerBase is the most base class.
handles IME related events.
TextInputHandlerBase and implements
NSTextInput protocol of Cocoa. Its
instance is created per
mozilla::widget::IMContextWrapper handles IME. The instance is created per top level window.