Hi,

not sure about the mode, but if there is already a selection, You can check it's type:

Selection[] selection = textArea.getSelection();

if (selection.length > 1) {
// multiple selection present!

} else if (selection.length == 1) {
if ((textArea.getSelectedLines().length > 1) && (! (selection[0] instanceof Selection.Range))) {
// single selection consists of more than one line, but is not of range type (hence rectangular!).
}
}