The Predictive Toxicology Challenge (PTC) for 2000-2001

C. Helma, R.D. King, S. Kramer and A. Srinivasan

NOTE:These pages are not maintained, they reflect the status of 2002.

Interested in the Challenge? email helma@informatik.uni-freiburg.de and we will keep you up to date

News

Dec 20 2001 Nov 7 2001

Contents

Predicting Chemical Carcinogens

Prevention of environmentally-induced cancers is a health issue of unquestionable importance.  Almost every sphere of human activity in an industrialized society faces potential chemical hazards of some form.  It is estimated that nearly 100,000 chemicals are in use in large amounts every day. A further 500-1000 are added every year.  Only a small fraction of these chemicals have been evaluated for toxic effects like carcinogenicity. The US National Toxicology Program (NTP)   contributes to this enterprise by conducting standardized chemical bioassays---exposure of rodents (mice and rats) to a range of chemicals--- to help identify substances that may have carcinogenic effects on humans.  However, obtaining empirical evidence from such bioassays is expensive and usually too slow to cope with the number of chemicals that can result in adverse effects on human exposure.  This has resulted in an urgent need for carcinogenicity models based on chemical structures and properties.  It is envisaged that such models would: The outcome of the bioassays conducted by the NTP has resulted in a large (by toxicological standards) database of compounds classified as carcinogens or otherwise. Predicting the outcome of these tests using chemical structure (and related information) presents a formidable test for techniques concerned with knowledge discovery from databases.  The Predictive Toxicology Challenge was devised to provide Machine Learning programs with the opportunity to participate in an enterprise of immense humanitarian and scientific value.

The Challenge for 2000-2001

The Challenge is to obtain models that predict the outcome of biological tests for the carcinogenicity of chemicals using information related to chemical structure only. Here we will require you to submit 4 models:
  1. Predicting outcome as POS or NEG on male rats;
  2. Predicting outcome as POS or NEG on female rats;
  3. Predicting outcome as POS or NEG on male mice; and
  4. Predicting outcome as POS or NEG on female mice
In each case, POS means that the chemical results in evidence of cancerous growth and NEG that there is no such evidence.

The timeframe for the  Challenge is as follows:
 

Stage Start Finish
Data Engineering September 01, 2000 March 01, 2001
Model Construction March 02, 2001 June 01, 2001
Model Evaluation June 02, 2001 August 15, 2001
Dissemination Early September 2001 Mid September 2001

Data Engineering (6 months starting Sep 01, 2000)

This phase of the Challenge will be concerned with:
  1. Calculation of chemical descriptors and feature construction.The initial chemical structures will be augmented by new chemical and structural features/background relations; and
  2. Data cleaning. Any errors in the data identified by participants will be corrected.
We encourage chemists, toxicologists and developers of programs for feature construction to participate in this phase by sending us new and potentially useful descriptors for the chemicals involved. To minimize bias during model construction, we have specified the rules for constructing these descriptors and the requirements for accompanying documentation. Submissions made here will be examined for toxicological relevance during the evaluation stage.

Model Construction (3 months starting Mar 02, 2001)

Models for the prediction of rodent carcinogenicity have to be based on the descriptors submitted in Stage I. Model constructors are free to choose all or parts of the dataset(s) and to use subsets of descriptors. Initial submissions (due to Jun02) will require  predictions for our independent test set containing approximately 200 compounds. From these predictions we will calculate the Receiver Operating Characteristic (ROC) for each submission. This will allow a cost-sensitive assessment of models in the evaluation stage. Structures, descriptors and instructions for submission will be provided at end of April. Based on this data, we will select a subset of optimal models in Phase III. Participants who submitted ``optimal'' models will have to provide a ``translation'' of their model, which is readable (and hopefully understandable) by toxicological experts.

Model Evaluation (2.5 months starting Jun 02, 2001)

From the ROC curves of all submissions we will identify a subset of models that are ``optimal'' for different cost conditions. Of these we will:
  1. Identify models that are particularly relevant to toxicology; and
  2. Identify any contributions made during the data engineering stage that were particularly informative
This will be a result of a co-operative exercise between the developers of the ``optimal'' models and our toxicologists (currently R. Benigni, Italian National Institute of Health, Italy; D.W. Bristol, NIEHS, USA; C. Helma, University Freiburg, Germany; and Y.T. Woo, EPA, USA).

Dissemination

A special workshop will be held on the Challenge as part of the ECML/PKDD Conferences at the University of Freiburg in September, 2001. This workshop will provide all participants with the opportunity to present their work. At this point, we expect to be in a position to decide if the submissions warrant a special journal publication.

Data

You can retrieve all data from

Initial Training Set

Attention: This is the original data from the NTP. No quality checks have been performed. 

Revised Training Set

Please report bugs, errors, inconsistencies, etc to cramer@informatik.uni-freiburg.de or helma@informatik.uni-freiburg.de. 

Structures for the Test Set

for Descriptors see Submissions from Stage I
ptc_testset.smi SMILES codes
ptc_testset_sdf.tar.gz MDL Molfiles
ptc_testset.pl.gz Prolog facts

Descriptors for the Training and Test Sets (Submissions from Stage I)

Several problems with descriptors for the training set have been fixed. Please use the new versions below for your final calculations.
Training Set Test Set Short Description Details Contributer
KULeuven.tgz KULeuvenFeatures.tgz functional groups, distances, ... KULeuven.txt Hendrik.Blockeel@cs.kuleuven.ac.be
393mol-839des-training-set.txt 337mol-839des-test-set.txt 839 molecular descriptors, calculated by DRAGON description-of-data.doc, Training-set-molecules.doc, Test-set-molecules.doc lucic@faust.irb.hr
TREYMERS.tar.gz, daylight_clusters.txt TREYMERS_VAL.tar.gz, daylight_clusters_val.txt molecular descriptors, calculated by different programs treymers.txt TREYMER1@JANBE.JNJ.COM
train.bci.dictionary, train.bci.fingerprints test.bci.fingerprints molecular substructures represented with fingerprints BCI_Frag.doc George.Cowan@pfizer.com
codesntp.txt, codecor.txt ptcodes.txt FCSS codes VINITI.txt, FCSSsm.rtf serge@viniti.ru
properties.tar.gz * test_properties.tar.gz molecular, atomic and bond properties calculated by various programs README.properties helma@informatik.uni-freiburg.de
fragments.tar.gz* test_fragments.tar.gz frequent linear fragments (substructures) README.fragments skramer@informatik.uni-freiburg.de
* Additional data provided by the organizers.

If you have questions concerning these datasets, please contact the authors directly.

Additional Data

added after the PTC

Instructions

Submissions in Stage I

Submissions in Stage II

PTC Workshop Submissions

Manuscripts for Bioinformatics

Results

Predictive Accuracy

Comprehensible Models

Optimal Models

Models that are on the convex-hull of the ROC curves
Model Group Contact
kwansei.pdf Kwansei okada@kwansei.ac.jp
leuven.pdf, leuven_notes.txt Leuven Hendrik.Blockeel@cs.kuleuven.ac.be
uta.pdf UTA holder@cse.uta.edu
viniti.zip VINITI serge@viniti.ru
waikato.pdf Waikato bernhard@cs.waikato.ac.nz

Near Misses

Models that are close to the convex-hull of the ROC curves
Model Group Contact
baurin_output.doc Baurin nicolas.baurin@univ-orleans.fr
sut_mr_rule.txt SUT kouyama@ia.noda.sut.ac.jp

Evaluation of correctly and incorrectly classified compounds

Workshop Proceedings

  1. Contents and Introduction
  2. First order models for the Predictive Toxicology Challenge 2001
  3. ILP-based Rule Induction for Predicting Carcinogenicity
  4. Predictive Toxicology using a Decision-tree Learner
  5. (The Futility of) Trying to Predict Carcinogenicity of Chemical Compounds
  6. Characteristic Substructures and Properties in the Chemical Carcinogenicity Studied by the Cascade Model
  7. Toxicology Analysis by Means of the JSM-method
  8. Delta-strong classification rules for characterizing chemical carcinogenicity
  9. Application of Graph-Based Concept Learning to the Predictive Toxicology Domain
  10. The application of several variable selection methods in the Predictive Toxicology Challenge 2000-2001
  11. Use of Learning Vector Quantization and BCI fingerprints for the Predictive Toxicology Challenge 2000-2001
  12. Carcinogenesis Modeling: Results Reflect Representation
  13. Predictive Toxicology Challenge (PTC) 2000-2001: A Toxicologist's View and Evaluation

Tools

Useful Links

helma@informatik.uni-freiburg.de