Win7 Math recognition

…you can look but you better not touch

Recently I’ve started working on a new project with my friend, so development of Falasol will be slower for a while. But fear not – I’m not going to give up on that. Falasol is my long term plan and what you can see so far is just a little part of it. I expect to get back to it next year. Meanwhile I’ll be occasionally posting stories from EqOCR development. You probably guessed that it will have something to do with equations & math – that is correct.

blackboard sexy

It was back at school when I discovered the charm of equations

Since me & my friend are big fans of Fringe science & cutting-edge technologies we were excited to find out about Win7 Math Handwriting Recognition capabilities. After going through API overview we couldn’t wait to plug into win7’s recognition engine and feed it with a basic quantum death ray formula. Until we figured out, that uhm… there is no such engine…

plan: analyze a sheet of handwritten equations



Turns out that you can analyze text written in whole bunch of languages: English, Polish, Chinese, Klingon… But in order to take advantage of Math Recognition, you need to use Math Input control. OK then, we thought, let’s embed the control, pass the handwriting and retrieve the analyzed output. Guess what… this control can’t be embedded.

Math Input control pop-up window

Math Input control pop-up window

Math Input control is so cool that no application is worthy hosting it. Math Input control can only be presented as a pop-up window. That would be fine for one or two lines, but our annihilation formula is one page long. And destroying the whole universe takes at least a dozen of calculation pages. Popping window for analyzing every single line? – no can do! Next idea: Let’s show the pop-up in a hidden state and pass the Ink written in our main window. Too bad this beauty won’t perform without audience… You need to call Show() prior to any other activities.

In the act of despair we considered displaying Math Input just for the while needed to programatically load the Ink & analyze it. Yeah, I know such flashing windows could cause epilepsy, but a true scientist should be used to all kinds of unexpected bursts and explosions. Yet another surprise… there is no GetResult() method present in the control’s API… That function is only provided via MCTFB (Manually Click The Freakin Button) event handler.

On top of that there is this shiny piece of COM object comes without any .net wrapper. And if you happen to use WPF you need to know that there are two versions of Ink object: full-blown-all-having COM Ink and it’s handicapped twin brother WPF Ink (with very limited set of methods, which – if you want to do any analysis – you need to convert to COM version anyway).

To resume: There only way to take advantage of Math Recognition in win7 is through a Math Input control. Which must be popped up right in the user’s face as a separate window. And requires manual confirmation whenever any analysis is done. I’m sure it does tremendous job with protecting the world from evil scientists (any scientists in fact), but you’ve got to do better than that to stop our crew :)


PS.

I do appreciate all the effort put in Math Input and it does a really good job in manual usage scenarios. This post was intended to show how frustrating it is not being able to use the engine programatically due to poor API support. I couldn’t find those limitations mentioned anywhere in the documentation, which gave the wrong impression about the tool’s capabilities (especially when compared to well designed regular handwriting recognition support).

Tags: ,

Reader's Comments »

  1. By Victor Hurdugaci on February 7, 2011 at 9:26 pm

    Cool stuff!
    Only one UI comment: the Cancel and Insert button don’t seem to have the ‘standard’ Windows button size (see Cancel/OK/Apply)

    By koshik on February 8th, 2011 at 6:41 pm

    Thanks :)
    I wish this was the only problem with that form…

  2. By kamil on February 7, 2011 at 11:21 pm

    Try to host it in a separate process running winform app and then use winapi to take ownership of the window/handle and render it within your app.

    By koshik on February 8th, 2011 at 6:51 pm

    Yeah, we probably could trick it into being rendered inside some winform host, but then it would look like a pop-up inside a window = terrible user experience. Still there is the ‘Insert’ button which need to be pressed manually. Since there is no shortcut key we would need to use some UI automation to simulate the click.
    These are way too many drawbacks and workarounds. We decided that this effort will be better invested in finding an alternative recognition method.