Research Interests:
Efforts in the development of NLP and [information technology] are converging on the recognition of the importance of some sort of corpus-based research as part of the infrastructure for the development of advanced language processing... more
Efforts in the development of NLP and [information technology] are converging on the recognition of the importance of some sort of corpus-based research as part of the infrastructure for the development of advanced language processing applications (Atkins, ...
Efforts in the development of NLP and [information technology] are converging on the recognition of the importance of some sort of corpus-based research as part of the infrastructure for the development of advanced language processing... more
Efforts in the development of NLP and [information technology] are converging on the recognition of the importance of some sort of corpus-based research as part of the infrastructure for the development of advanced language processing applications (Atkins, ...
Currently, the enormous span of social media usage – while providing valuable resources for linguistic behavior analysis – makes tracking and understanding these multilingual discussions a challenging task. We have undertaken a... more
Currently, the enormous span of social media usage – while providing valuable resources for linguistic behavior analysis – makes tracking and understanding these multilingual discussions a challenging task. We have undertaken a multidisciplinary comprehensive study of multilingual discussions via the development of specialized data collection techniques that discover and track multilingual users of social media, and their associated discussions, within a defined geographical region. To facilitate automatic discussion analysis of large numbers of discussions we generated a machine learning model based on ground truth data obtained from Amazon Turk. Our approach goes beyond analyzing social media posts in isolation, by analyzing them in the context of the discussion in which they appear. We show a selection of example discussions found using our approach which reveals a number of interesting socio-linguistic interactions in the communities that we sampled, in support of approach as a general methodology for multilingual community analysis.
Research Interests:
Automated systems such as information extraction (IE) pipelines are designed to facilitate situation awareness by providing human decision makers with relevant information, but beyond the validity of the pipeline itself, designing the... more
Automated systems such as information extraction (IE) pipelines are designed to facilitate situation awareness by providing human decision makers with relevant information, but beyond the validity of the pipeline itself, designing the output of the pipeline for optimal human understanding should be a goal. This paper presents results comparing comprehension of text documents with and without markup from a (simulated) IE pipeline in a simulated intelligence task. While previous work suggests that markup hurts both objective and subjective measures of performance and preference, this paper uses handgenerated markup designed to be maximally accurate and task relevant, finding more favorable results. These results, however, still point toward the limitations of markup and the importance of the task it is intended to facilitate.
Research Interests:
Taxonomy construction is not only a fundamental task for semantic analysis of text corpora, but also an important step for applications such as information filtering, recommendation, and Web search. Existing pattern-based methods extract... more
Taxonomy construction is not only a fundamental task for semantic analysis of text corpora, but also an important step for applications such as information filtering, recommendation, and Web search. Existing pattern-based methods extract hypernym-hyponym term pairs and then organize these pairs into a taxonomy. However, by considering each term as an independent concept node, they overlook the topical proximity and the semantic correlations among terms. In this paper, we propose a method for constructing topic taxonomies, wherein every node represents a conceptual topic and is defined as a cluster of semantically coherent concept terms. Our method, TaxoGen, uses term embeddings and hierarchical clustering to construct a topic taxonomy in a recursive fashion. To ensure the quality of the recursive process, it consists of: (1) an adaptive spherical clustering module for allocating terms to proper levels when splitting a coarse topic into fine-grained ones; (2) a local embedding module...
Research Interests:
Microblogging has become increasingly popular for commenting on current events, spreading gossip, and encouraging individualism--which favors its low-context communication channel. These social media (SM) platforms allow users to express... more
Microblogging has become increasingly popular for commenting on current events, spreading gossip, and encouraging individualism--which favors its low-context communication channel. These social media (SM) platforms allow users to express opinions while interacting with a wide range of populations. Hashtags allow immediate identification of like-minded individuals worldwide on a vast array of topics. The output of the analytic tool, Linguistic Inquiry and Word Count (LIWC)--a program that associates psychological meaning with the frequency of use of specific words-may suggest the nature of individuals’ internal states and general sentiments. When applied to groupings of SM posts unified by a hashtag, such information can be helpful to community leaders during periods in which the forming of public opinion happens in parallel with the unfolding of political, economic, or social events. This is especially true when outcomes stand to impact the well-being of the group. Here, we applied ...
Research Interests:
Actually using MT is not just a matter of buying a system, installing it, and feeding text through. Real-world text comes with spelling mistakes, bad grammar, missing portions, etc. But perhaps the users don't care, and just want a... more
Actually using MT is not just a matter of buying a system, installing it, and feeding text through. Real-world text comes with spelling mistakes, bad grammar, missing portions, etc. But perhaps the users don't care, and just want a rough idea of what the text is about. Aspects of the environment surrounding the system and the tasks for which the output is used can contribute significantly to the eventual success (or not) of the venture. This panel will share with the audience their experiences and advice on the real-world operational use of MT in the past, the present, and possibly their hopes for the future.
Research Interests:
Information Extraction (IE) research has made remarkable progress in Natural Language Processing using intrinsic measures, but little attention has been paid to human analysts as downstream processors. In one experiment, when participants... more
Information Extraction (IE) research has made remarkable progress in Natural Language Processing using intrinsic measures, but little attention has been paid to human analysts as downstream processors. In one experiment, when participants were presented text with or without markup from an IE pipeline, they showed better text comprehension without markup. In a second experiment, the markup was hand-generated to be as relevant and accurate as possible to find conditions under which markup improves performance. This experiment showed no significant difference between performance with and without markup, but a significant majority of participants preferred working with markup to without. Further, preference for markup showed a fairly strong correlation with participants’ ratings of their own trust in automation. These results emphasize the importance of testing IE systems with actual users and the importance of trust in automation.
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Expert networks are formed by a group of expert-professionals with different specialties to collaboratively resolve specific queries posted to the network. In such networks, when a query reaches an...
Research Interests:
Abstract: This study proffers two important findings:(1) automated machine translation (MT) evaluation is insensitive to the cognitive gravitas of proper names, contributing to its weak modeling of human judgments of higher quality MT... more
Abstract: This study proffers two important findings:(1) automated machine translation (MT) evaluation is insensitive to the cognitive gravitas of proper names, contributing to its weak modeling of human judgments of higher quality MT output, and (2) there is a" new" ...
Research Interests:
Abstract : This report documents the intersection of computational social network analysis and sociolinguistic research aimed at discovering how social intent is communicated through online bilingual speech acts in African cultures.... more
Abstract : This report documents the intersection of computational social network analysis and sociolinguistic research aimed at discovering how social intent is communicated through online bilingual speech acts in African cultures. Researchers from the US Army Research Lab (ARL) and Howard University (HU) exchanged information, data, and analyses to examine the feasibility of using automated text analytics software to provide contextual understanding within a text corpus. This effort extends the Army Research Office Partners in Research Transition program titled Extracting Social Meaning from Linguistic Structures Involving Code-Switching in English (and French) with Selected African Languages led by HU. It also provided test and evaluation opportunities for ARL prototype software designed to extract relational networks and sentiment from unstructured Tweets. This collaboration was driven by the realization that more social input is needed to refine context for sociolinguistic analysis and also by the increasing importance of modeling social issues for military decision making. To address social communication acts, we focus on using Twitter for sharing individual and collective opinions. Social media services in general have gained popularity in recent years and are frequently used for discovery and analysis of social intent. We examine the sociolinguistic features that can be used to discover social intent, discuss how social network analysis can be used to inform contextual nuances in which that intent is communicated, and describe how automated tools can be used to support sociolinguistic analysis. We conclude with future research directions that can extend the rich connections between computational social network analysis and the study of sociolinguistics.
Research Interests:
Real-time Analytics Platform for Interactive Data-mining (RAPID), a collaboration of University of Melbourne and Australia’s Defense Science and Technology Group (DSTG), consumes data streams, performs analytics computations, and produces... more
Real-time Analytics Platform for Interactive Data-mining (RAPID), a collaboration of University of Melbourne and Australia’s Defense Science and Technology Group (DSTG), consumes data streams, performs analytics computations, and produces high-quality knowledge for analysts. RAPID takes topic seed words and autonomously identifies emerging keywords in the data. Users direct the system, setting time-windowing parameters, thresholds, update intervals and sample rates. Apache Storm and Apache Kafka permit real-time streaming while logging options support off-line processing. Decision-support scenarios feature Commander Critical Information Requirements involving comparisons over time and time-sequencing of events, capabilities particularly well-served by RAPID technology, to be demonstrated in the presentation.
Research Interests:
The Human-Assisted Machine Information Exploitation (HAMIE) investigation utilizes large-scale online data collection for developing models of information-based problem solving (IBPS) behavior in a simulated time-critical operational... more
The Human-Assisted Machine Information Exploitation (HAMIE) investigation utilizes large-scale online data collection for developing models of information-based problem solving (IBPS) behavior in a simulated time-critical operational environment. These types of environments are characteristic of intelligence workflow processes conducted during human-geo-political unrest situations when the ability to make the best decision at the right time ensures strategic overmatch. The project takes a systems approach to Human Information Interaction (HII) by harnessing the expertise of crowds to model the interaction of the information consumer and the information required to solve a problem at different levels of system restrictiveness and decisional guidance. The design variables derived from Decision Support Systems (DSS) research represent the experimental conditions in this online single-player against-the-clock game where the player, acting in the role of an intelligence analyst, is tasked with a Commander’s Critical Information Requirement (CCIR) in an information overload scenario. The player performs a sequence of three information processing tasks (annotation, relation identification, and link diagram formation) with the assistance of ‘HAMIE the robot’ who offers varying levels of information understanding dependent on question complexity. We provide preliminary results from a pilot study conducted with Amazon Mechanical Turk (AMT) participants on the Volunteer Science scientific research platform.
Research Interests:
Research Interests:
Research Interests:
Expert networks are formed by a group of expert-professionals with different specialties to collaboratively resolve specific queries posted to the network. In expert networks, decentralized search, operating purely on each expert's... more
Expert networks are formed by a group of expert-professionals with different specialties to collaboratively resolve specific queries posted to the network. In expert networks, decentralized search, operating purely on each expert's local information without any knowledge of network global structure, represents the most basic and scalable routing mechanism. However, there is still a lack of fundamental understanding of the efficiency of decentralized search. In this regard, we investigate decentralized search by quantifying its performance under a variety of network settings. Our key findings reveal that under certain network conditions, decentralized search can achieve significantly small query routing steps (i.e., between O(log n) and O(log2 n), n: total number of experts in the network). To the best of our knowledge, this is the first work studying fundamental behaviors of decentralized search in expert networks.
Research Interests:
Within operational environments decisions must be made quickly based on the information available. Identifying an appropriate knowledge base and accurately formulating a search query are critical tasks for decision-making effectiveness in... more
Within operational environments decisions must be made quickly based on the information available. Identifying an appropriate knowledge base and accurately formulating a search query are critical tasks for decision-making effectiveness in dynamic situations. The spreading of graph data management tools to access large graph databases is a rapidly emerging research area of potential benefit to the intelligence community. A graph representation provides a natural way of modeling data in a wide variety of domains. Graph structures use nodes, edges, and properties to represent and store data. This research investigates the advantages of information search by graph query initiated by the analyst and interactively refined within the contextual dimensions of the answer space toward a solution. The paper introduces SLQ, a user-friendly graph querying system enabling the visual formulation of schemaless and structureless graph queries. SLQ is demonstrated with an intelligence analyst information search scenario focused on identifying individuals responsible for manufacturing a mosquito-hosted deadly virus. The scenario highlights the interactive construction of graph queries without prior training in complex query languages or graph databases, intuitive navigation through the problem space, and visualization of results in graphical format.
Research Interests:
Abstract : Our technique for selecting foreign-language technical terms for human-use glossaries and automatic processor lexicons offers a customized solution based on sound principles that has resulted in an effectiveness breakthrough... more
Abstract : Our technique for selecting foreign-language technical terms for human-use glossaries and automatic processor lexicons offers a customized solution based on sound principles that has resulted in an effectiveness breakthrough for the Army. Prepared in support of a single Joint Task Force (JTF), its principle-based underpinnings justify its use in similar applications for various JTFs involved in strategic operations. Language induces expectations from its community of use that can be exploited to provide more effective machine translation (MT). Entries in finely tuned glossaries and lexicons, which are devoid of ambiguity, carry a valence that activates readers' associated world knowledge. The clarity of the entry builds reader confidence while the activated semantic fields permit readers a higher likelihood of accurate context interpretation than would otherwise be possible. A glossary prepared for one document in particular serves to illustrate the method employed throughout the project for glossary development. This document and a human translation are freely available at http://alep.stanford.edu/.
Research Interests:
Research Interests:
Abstract : The integrated development environment of the General Architecture for Text Engineering (GATE), or GATE Developer, is used to annotate entities in a text document consisting of messages in and around the Baghdad area (SYNCOIN... more
Abstract : The integrated development environment of the General Architecture for Text Engineering (GATE), or GATE Developer, is used to annotate entities in a text document consisting of messages in and around the Baghdad area (SYNCOIN data). Highlighting entities, such as person(s), location(s), and organization(s), may result in a more structured format for faster comprehension of the data. The application for entity determination is called a nearly-new information extraction, or ANNIE: a system of seven processing resources (PRs) in GATE. ANNIE is executed from the graphical user interface (GUI). Other PRs, such as those for machine learning, and the capability for user-defined applications are managed as a collection of reusable objects for language engineering (CREOLE); an icon for the CREOLE plug-in manager exists at the GUI as well.
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Recently, there has been an emphasis on creating shared resources for natural language processing applications. This has resulted in the development of high-quality tools and data, which can then be leveraged by the research community as... more
Recently, there has been an emphasis on creating shared resources for natural language processing applications. This has resulted in the development of high-quality tools and data, which can then be leveraged by the research community as components for ...
Research Interests:
Mining entity synonym sets (i.e., sets of terms referring to the same entity) is an important task for many entity-leveraging applications. Previous work either rank terms based on their similarity to a given query term, or treats the... more
Mining entity synonym sets (i.e., sets of terms referring to the same entity) is an important task for many entity-leveraging applications. Previous work either rank terms based on their similarity to a given query term, or treats the problem as a two-phase task (i.e., detecting synonymy pairs, followed by organizing these pairs into synonym sets). However, these approaches fail to model the holistic semantics of a set and suffer from the error propagation issue. Here we propose a new framework, named SynSetMine, that efficiently generates entity synonym sets from a given vocabulary, using example sets from external knowledge bases as distant supervision. SynSetMine consists of two novel modules: (1) a set-instance classifier that jointly learns how to represent a permutation invariant synonym set and whether to include a new instance (i.e., a term) into the set, and (2) a set generation algorithm that enumerates the vocabulary only once and applies the learned set-instance classifi...
Research Interests:
The DARPA MT evaluations of the early 1990s, along with subsequent work on the MT Scale, and the International Standards for Language Engineering (ISLE) MT Evaluation framework represent two of the principal efforts in Machine Translation... more
The DARPA MT evaluations of the early 1990s, along with subsequent work on the MT Scale, and the International Standards for Language Engineering (ISLE) MT Evaluation framework represent two of the principal efforts in Machine Translation Evaluation (MTE) over the past decade. We describe a research program that builds on both of these efforts. This paper focuses on the selection of MT output features suggested in the ISLE framework, as well as the development of metrics for the features to be used in the study. We define each metric and describe the rationale for its development. We also discuss several of the finer points of the evaluation measures that arose as a result of verification of the measures against sample output texts from three machine translation systems.
Research Interests: Computer Science, Artificial Intelligence, Natural Language Processing, Assessment, Computational Linguistics, and 11 moreMachine Translation, Metrics, Applied Linguistics, Evaluation, Performance, Standardization, Measure, Standardisation, Research Project, Multi Dimensional, and Language Engineering
Research Interests:
Tasks performed on machine translation (MT) output are associated with input text types such as genre and topic. Predictive Linguistic Assessments of Translation Output, or PLATO, MT Evaluation (MTE) explores a predictive relationship... more
Tasks performed on machine translation (MT) output are associated with input text types such as genre and topic. Predictive Linguistic Assessments of Translation Output, or PLATO, MT Evaluation (MTE) explores a predictive relationship between linguistic metrics ...
Research Interests:
Scaling the ISLE Taxonomy: Development of Metrics for the Multi-Dimensional Characterisation of Machine Translation Quality Keith J. Miller, Michelle Vanni ... scores for MT output to a set of the same tests' scores for... more
Scaling the ISLE Taxonomy: Development of Metrics for the Multi-Dimensional Characterisation of Machine Translation Quality Keith J. Miller, Michelle Vanni ... scores for MT output to a set of the same tests' scores for naturally-occurring target language text (Jones and Rusk 2000 ...
Research Interests: Computer Science, Artificial Intelligence, Natural Language Processing, Assessment, Computational Linguistics, and 11 moreMachine Translation, Metrics, Applied Linguistics, Evaluation, Performance, Standardization, Measure, Standardisation, Research Project, Multi Dimensional, and Language Engineering
Research Interests:
Research Interests:
Research Interests:
Machine Translation evaluation has been more magic and opinion than science. The history of MT evaluation is long and checkered - the search for objective, measurable, resource-reduced methods of evaluation continues. A recent trend... more
Machine Translation evaluation has been more magic and opinion than science. The history of MT evaluation is long and checkered - the search for objective, measurable, resource-reduced methods of evaluation continues. A recent trend towards task-based evaluation inspires the question - can we use methods of evaluation of language competence in language learners and apply them reasonably to MT evaluation? This paper is the first in a series of steps to look at this question. In this paper, we will present the theoretical framework for our ideas, the notions we ultimately aim towards and some very preliminary results of a small experiment along these lines.