[Langutils-devel] Period not correctly tokenized?

[Jianshi Huang]

Hi Ian,

On Wed, Oct 12, 2011 at 1:01 AM, Ian Eslick <eslick at media.mit.edu> wrote:
> Periods are handled specially because they show up in numbers, abbreviations, e.g. and i.e., etc.  You lose numbers as tokens if you split out periods naively.

I see. But it seems there's a bug in the handling.

Please ignore my patch.


-- 
黄　澗石 (Jianshi Huang)
http://huangjs.net/