Tasks‎ > ‎

BB3

Bacteria Biotope - Event extraction of microorganisms and habitats with ontologies and their linking



Downloads
 Entity categorization
cat
Training
Development
Test
     + named-entity recognition    
cat+ner Training Development  Test
 Event extraction
event
Training Development  Test
   + named-entity recognition     event+ner Training Development  Test
 Knowledge base population
kb
Training Development  Test
      + named-entity recognition     kb+ner Training Development  Test
OntoBiotope habitat ontology (OBO)
NCBI Taxonomy from 2016-02-03 (33Mb)
   

 
 

Information Extraction Goal 

1. Promote information extraction on microorganism biodiversity.

2. Assess the performance of automatic categorization and relation extraction systems.  

Motivation in biology 

Bacteria biotope is a critical information for studying the interaction mechanisms of the bacteria with their environment from genetic, phylogenetic and ecology perspectives. The information on habitats where bacteria live is a particularly critical in applied microbiology such as food processing and safety, health sciences and waste processing. There is also fundamental research that requires this knowledge like metagenomics studies and phylogeography/phyloecology. 

No database supplies the habitats of bacteria in a comprehensive and normalized way. This information is spread in free texts over articles and databases. Bacteria biotopes are described in large sets of free texts, scientific papers and numerous databases, such as databases of sequences (e.g. SRA, GenBank, GOLD), biological sample banks, collections (e.g. ATCC, DSMZ) and biodiversity surveys (e.g. GBIF). 

IE goal 

To fulfill these needs, as in previous Shared Tasks in 2011 and 2013 the IE systems should be able to: 1. detect mentions of habitats and species;
 2. categorize them with large ontologies;
 3. extract events between bacteria and their habitats. Once the habitats are identified, they must be normalized to be compared. The OntoBiotope ontology is used for microorganism biotope description, it has been successfully used in previous BB tasks.  The well-recognized resource NCBI taxonomy is used for organism classification and normalization.









The concept "seedling" of the "growing plant part of OntoBiotope is used to tag "axenic rice seedling" in the text.

Events, entities and categories of BB task in AlvisAE editor.

Representation and Task setting

BB3 corpus annotation follows BioNLP-ST2013 representation with a few modifications in order to cope with discontinuous entities and categorization with concepts from an ontology. An updated version of the habitat part of OntoBiotope is provided to participants in OBO format. It contains concepts organized in a hierarchy of is-a relations. Ontobiotope concepts may have more than one parent and the concept labels have synonyms. The NCBI taxonomy can be downloaded here.

Entities

The BB task includes three types of entities, bacteria, habitats and geographical places.

Bacteria

Bacteria entities are annotated as contiguous spans of text that contains a full unambiguous prokaryote taxon name, the type label is Bacteria. The Bacteria type is a taxon, at any taxonomic level from phylum (Eubacteria) to strain. The category that the text entities have to be assigned to is the most specific and unique category of the NCBI taxonomy resource. In case a given strain, or a group of strains is not referenced by NCBI, it is assigned with the closest taxid in the taxonomy.

Habitat

Habitat entities are annotated as spans of text that contains a complete mention of a potential habitat for bacteria, the type label is Habitat. Habitat entities are assigned one or several concepts from the habitat subpart of the OntoBiotope ontology. The assigned concepts are as specific as possible. OntoBiotope defines most relevant microorganism habitats from all areas considered by microbial ecology (hosts, natural environment, anthropized environments, food,  medical, etc.). Habitat entities are rarely referential entities, they are usually noun phrases including properties and modifiers. There are rare cases of habitats referred with adjectives or verbs. The spans are generally contiguous but some of them are discontinuous in order to cope with conjunctions.

Geographical

 Geographical entities are geographical and organization places denoted by official names.

Events

The BB task considers a single type of event.

Lives_In

The Lives_in event has two mandatory arguments, the bacterium and the location where it lives (either an Habitat or a Geographical entity). Whenever the habitat is host part and the host is mentioned in the text, both habitats participate in two distinct events. The Lives_In event does not include trigger words.

Evaluation and criteria 

We propose three sub-tasks with two modalities each. Each sub-task has a plain modality where named-entities are given as input, thus participants are not required to perform named-entity recognition. In the second modality, named-entities are not provided, thus participant methods must perform named-entity recognition and submissions will be partly evaluated on their boundary accuracy.

Examples for each task here.

1.  Bacteria and habitat detection and categorization (BB-cat and BB-cat+ner)

Participant systems are evaluated for their capacity to categorize two kind of entities, Bacteria and Habitat with the NCBI Taxonomy taxa and OntoBiotope habitat concepts respectively.

Input: document texts, title and paragraph segmentation,  NCBI Taxonomy and OntoBiotope habitat ontologyEntities (BB-cat),

To be predicted: Bacteria and Habitat entity categories. Entity boundaries (BB-cat+ner). 

Evaluation

The evaluation will focus on the accuracy of the predicted categories compared to gold reference. The measures will be similar to BB13. A concept distance measure has been designed in order to sanction over-generalization or over-specialization with a fair penalty. Note that if an entity has several categories, then it is a conjunction: all categories must be predicted.

For cat+ner, boundary accuracy will be factored in the evaluation since the inclusion or exclusion of modifiers can change the meaning and the categorization of phrases.

2.  Entity and event extraction (BB-event and BB-event+ner)

Participant systems are evaluated for their capacity to extract Lives_In events among Bacteria, Habitat and Geographical entities.

Input: document texts, title and paragraph segmentation. Bacteria, Habitat and Geographical entities (BB-event)

To be predicted:  Lives_In events. BacteriaHabitat and Geographical entities (BB-event+ner).

Evaluation

The evaluation measures will be Recall and Precision of predicted events against gold events. Gold coreferences will be used to factor equivalent relations.

For event+ner, boundary accuracy will be factored in the evaluation.


2.  Knowledge Base extraction (BB-kb and BB-kb+ner)

Participant systems are evaluated for their capacity to build a knowledge base from the corpus. 
The knowledge base is the set of Lives_in events with the categories of their Bacteria and Habitat arguments. This sub-task is a sum of sub-tasks cat and event, since participants must predict entities categorizations and Lives_In events.

The goal of the task is to measure how much of the information content of the corpus can be extracted automatically.

 Input: document texts, title and paragraph segmentation. Bacteria and Habitat entities (BB-kb)

To be predicted: Bacteria and Habitat entity categories, Lives_In events. Bacteria and Habitat entities (BB-kb+ner). 

Evaluation

The evaluation measures will be Recall and Precision of predicted events against gold events.

For cat+ner, boundary accuracy will be factored in the evaluation.


Corpus

Documents

The corpus is composed of scientific paper abstracts. In previous BB tasks, the corpus was composed of general purpose texts (i.e. mostly web pages of genomics projects) that were easily understandable by task participants. The scientific papers abstracts are more useful sources of detailed and scientific information for biologists. The corpus is a subset of a corpus of 1,16 million PubMed references that mention bacteria and habitats. 215 documents have been automatically selected in order to reflect the diversity of the habitats.

Annotation

The manual annotation was performed by 7 annotators of Bibliome group from the biology (2) and computer science (4) domains in a double-blind way after an automatic preannotation by the Alvis Suite. The annotators have used AlvisAE editor for annotation and adjudication.

History of the task

The three subtasks are similar to BB previous tasks (BB'11, BB'13), except for the knowledge base task and taxon normalization. The official results of BB'13 were: 0.66 SER for habitat detection and categorization and 0.14 F1-measure on event and entity extraction. More recent results may be found in [BMC Bioinformatics, vol 16, supp. 10 and 16].

Results on the new taxon categorization task has been reported in previous work notably about the SPECIES and ORGANISM software[1], which achieved over 85% F-measure.

Contact

Robert Bossy
Bibliome-MaIAGE
Inra, France
e-mail: robert (dot) bossy (at) jouy (dot) inra (dot) fr


[1] The SPECIES and ORGANISMS Resources for Fast and Accurate Identification of Taxonomic Names in Text. Pafilis E, Frankild SP, Fanini L, Faulwetter S, Pavloudi C, et al. (2013) PLoS ONE 8(6): e65390