Amazon AI Researchers Release a Dataset of 400,000 Transliterated Names To Aid the Development of Natural-Language-Understanding Systems

New submitter georgecarlyle76 writes: Amazon AI researchers have publicly released a dataset of almost 400,000 transliterated names, to aid the development of natural-language-understanding systems that can search across databases that use different scripts. They describe the dataset's creation in a paper [PDF] they're presenting at COLING, together with experiments using the dataset to train different types of machine learning models.

Pretty amazing

[110010001000]

It is really amazing all the research they are able to do there. I would have thought the humidity and rain would wreak havoc with computers. Maybe it helps there is no Internet access as well, so they aren't distracted by social media and can focus on AI research.

Re:

[Tablizer]

All because they run Linux instead of Windows.

Not the usual NN/ML hype paper

[isj]

The paper is informative. They point out the obvious problems (translation from scripts/orthography missing vowels, but also that many names are actually quite rare. In their dataset 73% of the names only occur once.

They also compare the results with traditional hardcoded rules, and find that neural networks may not be better.So kudos for including non-positive results in the paper.

What do these sentences mean?

[dhaen]

"In most names, the pronunciation of the last name is independent of the rst or middle names"

"So it makes sense to train a transliteration system on independent pairs of first names, last names, and so on."

I'm confused about the meaning of the sentences above. There seems to be an emphasis on last names. Now as an English speaker that sounds ok, but since this about multiple languages where often it's family name first, it doesn't seem to compute.

Re:

[isj]

What they mean is that there is no or nearly no correlation between first name and last name.
So John, Bob, Rob, Randy, Elizabeth, Maggie are all equally likely for surname X.

Of course there will be a weak correlation if the surname is Fleischer then the first name has a slightly higher probability of being Jens, Uwe or Reichard.