Google AI Blog: UI

Grounding Natural Language Instructions to Mobile UI Actions

Friday, July 10, 2020

Posted by Yang Li, Research Scientist, Google ResearchMapping Natural Language Instructions to Mobile UI Action SequencesACL 2020automatic action sequence mappingGitHub repositoryConstructing Language Grounding Modelslanguage groundingaction phrase-extractiongrounding

The workflow of grounding language instructions to executable actions.

Transformerarea attention

The action phrase extraction model takes a word sequence of a natural language instruction and outputs a sequence of spans (denoted in red boxes) that indicate the phrases describing the operation, the object and the argument of each action in the task.

The grounding model takes the extracted spans as input and grounds them to executable actions, including the object an action is applied to, given the UI screen at each step during execution.

Resultspublic android UI corpusgraph convolutional networkfeedforward networkConclusionAcknowledgements Many thanks to my co-authors Jiacong He, Xin Zhou, Yuan Zhang and Jason Baldridge on this work at Google Research. I would also like to thank Gang Li who provided generous help for creating open-source datasets, and Ashwin Kakarla, Muqthar Mohammad and Mohd Majeed for their help with the annotations.

Build your own Machine Learning Visualizations with the new TensorBoard API

Monday, September 11, 2017

Posted by Chi Zeng and Justine Tunney, Software Engineers, Google Brain Teamwe open-sourced TensorFlow in 2015TensorBoardexploring clusters in high-dimensional spacesset of APIstensorboard/plugins directory on GitHubprecision-recall curves

A TensorFlow summary op used to collect data for later visualization. [GitHub]

A Python backend that serves custom data. [GitHub]

A dashboard within TensorBoard built with TypeScript and polymer. [GitHub]

provides a demoTensorBoard “Greeter” pluginexisting pluginsBeholder,Chris Andersonhere

issue trackerAcknowledgements

Harness the Power of Machine Learning in Your Browser with Deeplearn.js

Friday, August 11, 2017

Posted by Nikhil Thorat and Daniel Smilkov, Software Engineers, Google Big Picture Teamobject recognitionlanguage translationhealthPAIRdeeplearn.js 0.1.0

Andrej Karpathy's convnetjsTensorFireTensorFlowNumPywithout writing a single line of code

deeplearn.js homepage

How to measure translation quality in your user interfaces

Friday, October 30, 2015

Posted by Javier Bargas-Avila, User Experience Research at Google200 languages that are spoken by at least 3 million people

Three versions of the YouTube UI: (a) the original, (b) YouTube without text elements, and (c) YouTube without graphic elements. It gets apparent how the textless version is stripped of the most useful information: it is almost impossible to choose a video to watch and navigating the site is impossible.

Measuring user rated language quality: Development and validation of the user interface Language Quality Survey (LQS)International Journal of Human-Computer Studiesautomatiqueautomobilefull survey can be found in the publicationCase study: applying the LQS in the field

The data revealed that about one third of the languages showed subpar language quality levels, when compared to the source language.

To understand the source of these problems and fix them, we analyzed the qualitative feedback users had provided (every time someone selected the lower two end scale points, pointing at a problem in the language, a text box was surfaced, asking them to provide examples or links to illustrate the issues).

The analysis of these comments provided linguists with valuable feedback of various kinds. For instance, users pointed to confusing terminology, untranslated words that were missed during translation, typographical or grammatical problems, words that were translated but are commonly used in English, or screenshots in help pages that were in English but needed to be localized. Some users also pointed to readability aspects such as sections with old fashioned or too formal tone as well as too informal translations, complex technical or legal wordings, unnatural translations or rather lengthy sections of text. In some languages users also pointed to text that was too small or criticized the readability of the font that was used.

In parallel, in-depth expert reviews (so-called “language find-its”) were organized. In these sessions, a group of experts for each language met and screened all of YouTube to discover aspects of the language that could be improved and decided on concrete actions to fix them. By using the LQS data to select target languages, it was possible to reduce the number of language find-its to about one third of the original estimation (if all languages had been screened).

Reflections on Digital Interactions: Thoughts from the 2012 NA Faculty Summit

Thursday, August 2, 2012

Posted by Alfred Spector, Vice President of Research and Special InitiativesComputer Science Faculty SummitprogramKnowledge GraphThis pagePower Searching Coursetheir blog

Suggesting a Better Remote Control

Thursday, November 4, 2010

Posted by Ullas Gargi and Rich Gossweiler, Research TeamQuickSuggest.

Nov	DEC	Jan
	20
2019	2020	2021

Blog

Grounding Natural Language Instructions to Mobile UI Actions

Build your own Machine Learning Visualizations with the new TensorBoard API

Harness the Power of Machine Learning in Your Browser with Deeplearn.js

How to measure translation quality in your user interfaces

Reflections on Digital Interactions: Thoughts from the 2012 NA Faculty Summit

Suggesting a Better Remote Control

Labels

Archive

Feed