Challenge Design and Participation
Master students prepare mini-AI challenges in groups of 6 people, which are then solved by other students.
Mini Challenges 2022-2023
Challenges
Abstract
Task
FAIR UNIVERSE
Particle physics
High energy physicists at CERN use simulations in order to reproduce collisions that occurs in the Large Hadron Collider (LHC). Number of particles created in a single collision can range from a few to several hundred. Once they developped a theory which predicts the existence of a new particle, physicists run simulations and seek evidence of new particle. To do so, they classify all the particles resulting from a collision between background particles (uninteresting ones that they already know) and signal particles (the ones they are interested in).
The Fair Universe challenge is a toy-example for this problem. Instead of working in a high-dimensional feature space, we consider 2D points (2-dimensional feature vectors) that belongs either to signal or background class. The aim is to build a model that classifies them correctly.
The difficulty is that data from simulations may be biased (unfair). The task of the participants is to achieve good classification in the presence of bias/systematic errors.
Bias in data
Over the past few years, society has been grappling with the extent to which human biases can be incorporated into artificial intelligence systems, leading to harmful consequences. With the proliferation of big data, optimizing AI systems solely for performance will not only fail to improve real-world applications, but also increase the risk of bias in ways that may not be immediately apparent.
This is a computer vision challenge. The goal of this challenge is to perform multi-task classification with a training dataset biased towards Styles. You will need to make sure it performs well on an unbiased test dataset. More info can be found in Data tab.
Causal discovery
Understanding the causal relationships between variables is a central goal in many fields, ranging from the social and behavioral sciences to the natural and physical sciences. It is especially common in medical studies, as understanding the causal relationship between genetic or environmental factors and the occurrence or severity of a disease is the first step in developing strategies for prevention, diagnosis and treatment. The latest fairness principles are grounded in causality, acknowledging the widespread belief that incorporating causality is imperative in effectively addressing the problem of fairness.
The goal of this challenge is to recover the causal direction of the relationship between two variables A and B. 4 scenarios are possible:
A causes B
B causes A
A and B are both consequences of a common cause C
A and B are independant
This is a computer vision challenge. The goal of this challenge is to perform multi-task classification with a training dataset biased towards Styles. You will need to make sure it performs well on an unbiased test dataset. More info can be found in Data tab.
Style transfer
This challenge focused on addressing bias in image datasets, where the background is a common source of bias. For example, classifiers may be induced to rely on background information such as airplanes in the sky, cars on roads, or boats on the water. By using "style transfer" to create a fake relationship in which the style of the painting correlates with the subject that is to be recognized. the project will intentionally introduce biases into the dataset, providing an opportunity for researchers to develop classifiers that can recognize subjects accurately despite these biases. The ultimate goal is to create a dataset that can be used to train classifiers that are immune to bias, and thus more robust and reliable for real-world applications.
The problem is a multiclass classification problem. The dataset contains 3 classes of (512x512) RGB images, and each class contains 20 images. For each class, their is 3 styles of images with one style being dominant per class to induce some bias.
You are given 60 images for training as well as their labels in a csv file. You must design a model that predicts the class of each images while taking into consideration rate groups within each class. The evaluation metric gives more weight to the rare groups.
Mini Challenges 2021-2022
Challenges
Abstract
Task
Living species classification problem
Classifying living species is a crucial problem. It provides a way of identifying different groups of organisms and yields an internationally recognized way of referring to organisms. It helps in quantifying biodiversity because when organisms can be identified, population changes can be monitored. This helps in conservation of living organisms. It enables scientists to explain how different organisms are related to each other.
This is a multi-class image classification problem. The challenge protocol is in the AutoML setting: code is submitted and blindly evaluated for training and testing on an unknown dataset (a different one in each phase). Sample data and a starting kit are provided to prepare the code.
Problems of bias, particularly due the image background, will ha e to be studied.
Histology classification
You will help histologist and histopathologist to make better diagnosis, using images of microscope slices. This problem is important to automate the process of medical analyses and help study problems of bias due to differences in staining, sample contamination, lighting, etc.
This is a multi-class image classification problem. As the previous one, this challenge is designed to be an automatic machine learning challenge. The principle is to build more robust models, able to fit different datasets within the same domain. You will have to make sure that your model is able to adapt to datasets it has never seen before, by automatically adapting its hyperparameters, avoiding overfitting and other ills that data scientists know so well.
Trust us, it's AI
We address directly the bias in machine learning models against groups in society. Inspired by the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) biased software, this challenge deals with a classification problem that is adjudicating on the suspect based on criminal activities.
This is a multi-class classification challenge from tabular data, using a classical challenge protocol with a single dataset split into training/validation/test sets. It is with code submission. The code is evaluated on validation data in phase 1 and on test data in phase 2. The aim of this challenge is not to replace the judge in the court, but to raise awareness among participants about the subject of fairness in AI and encourage them to put more effort into tackling the potential bias and discrimination in machine learning models in favor of or against certain groups.
Mini Challenges 2020-2021
Challenges
(M1, Project A)
Abstract
Task
Detect Pneumonia fro Chest X-rays
Among the medical imaging examinations, chest X-rays are the most common. Each year, there are 3.6 billion medical procedures involving ionizing radiation of which over 2 billion are chest X-rays. Pneumonia is one among the deadly diseases that chest X-ray can help to detect. Help automatic interpretation of chest X-ray to detect pneumonia.
Originally this [competition site] was created using CNN features, but is was biased. It was replaced by the cropped image competition.
The problem a binary classification task of normal vs pneumonia. You will have to predict the patients are normal or has pneumonia from their chest X-rays. Additionally, you may need to predict where in the image the lung anomaly is located. The data consist of chest X-ray images preprocessed to CNN features (in this version of the challenge; to be updated). ??
Urban growth prediction
The goal of this project is to find a minimal set of socio-economic indicators that can explain to a certain degree the population growth in Urban areas. The winning model can help governments to create better policies to increase wellness and regulate urban growth to avoid problems brought by either stagnation and overgrowth of urban areas.
The problem is a regression problem (predict urban growth from socio-economic indicators, and a feature selection problem (select a small feature subset to build the predictor).
There are 2 challenge versions:
As a regression problem only.
As a regression + feature selection problem.
Deep pollination
Insects, specially pollinating insects have a great role in the biodiversity in the nutrient cycling and the functioning of ecosystems. One of the the main threats to our ecosystem is the continuous decline in the population of various pollinating insects. Help monitoring the population of pollinating insects by classifying images taken by volunteer.
The problem is a multiclass classification problem. The aim of the challenge is to classify images of insects into 5 categories: Bee, Wasp, Butterfly, Other Insect, Other.
There are 2 challenge versions:
From preprocessed data as KAZE features.
From raw image data.
In both cases you must submit code, but for the second version, you must submit pre-trained models.
[SAMPLE CHALLENGE]
Iris classification sample competition
This is the well known Iris dataset from Fisher's classic paper (Fisher, 1936). The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. One class is linearly separable from the other 2; the latter are NOT linearly separable from each other.
The problem is a multiclass classification problem. Each sample (an Iris) is characterized by its sepal and petal width and length (4 features). You must predict the Iris categories: setosa, virginica, or versicolor.
Mini Challenges 2019-2020
Challenges
(M2 students)
Abstract
Task
L2 groups
(who solved it)
Plankton recognition
A lot of species are endangered, if not simply disappearing.Help protect what can still be protected, and save what can still be saved.
Our planet is covered at 70% by water, which makes it obvious that a great part of the diversity that can be found on Earth will be found here, under the water. Help monitoring the health of the oceans by studying planktons (microscopic animals) and determining their diversity in a certain location, as a measure of biodiversity.
The problem here is a multiclass classification problem. Each element is a plankton photography. Originally, each image contained 300x300 pixel, but we preproced and reduced them to 100x100 pixel for memory reason. We then used the vertical and horizontal histograms as well as the mean and the variance as features.
Each image is then characterized by these 202 features.
Traffic prediction
Lemonade sales are dependent on car traffic near your place of business. Your mission, should you decide to accept it, is to predict the number of cars that will pass by at a given date, hour, and additional meteorological informations, near the lemonade stand.
This is a regression problem. You have to predict highway traffic volume as a function of 58 features (including specification about time and different weather descriptions).
Malaria parasite identification
Malaria is a disease that is spread by a bite of an infected female mosquito, caused by the Plasmodium, genus of parasites, transmitted by mosquito bites. Early diagnosis could help treat and control the disease. In this challenge you will have access to pre-processed images of segmented cells from the thin blood smear slide. Your goal is to detect parasitized cell images from uninfected ones in order to diagnose malaria.
The problem is a binary classification problem. Each sample is an an image of a cell which can be infected or not. You are given for training a set of images which is reduced to 6 numerical features, so you have a matrix containing those 6 features for 60% of the total number of images.
Acknowledgements: These challenges are hosted by CodaLab. We received a grant of the FCS Paris-Saclay and sponsorship of Microsoft Azure for Research and Google Research.
Mini Challenges 2018-2019
Challenges
(M2 students)
Abstract
Task
L2 groups
(who solved it)
Areal
Recognize landscapes from satellite images
This challenge views things from high! From areal images you'll figure out whether it is a beach, chaparral, cloud, desert, forest, island, lake, meadow, mountain, river, sea_ice, snowberg, or wetland. Classifying terrains is important to control urban development, favor economic growth, and protect environment. [DIAPOS]
The problem is a multi-class classification problem. You must predict the categories of 13 classes.
There are two possible challenges: one from raw data and one from preprocessed data. In its raw data version, the challenge is to classify images characterized by 128*128 pixel maps. In its preprocessed data version, the challenge is to classify vectors of 4096 high-level abstract features extracted with a pre-trained CNN.
Health data challenge
The dataset is a set of patients which have been diagnosed at different stages of cancer. Your task is to improve the classification results regarding the stages of those patients. [DIAPOS]
This is a multi-class classification problem. You must classify cancer stages among a specific population in one of 10 categories. The data is a matrix of (number of patients) lines * (number of features per patient) columns. Features correspond to methylation information related to the medical condition of each patient.
Learning to run a power network
The goal of this challenge is to control electricity transportation in power grids, while keeping people and equipment safe. This is the "gamification" of a serious problem: operating the grid is becoming increasingly complex because of the advent of less predictable renewable energies, the globalization of energy markets, growth in consumption and concurrent limitations on new line construction. [DIAPOS]
This is a reinforcement learning (RL) problem. You will have access to a simulator of a small scale grid. The designed RL agents should learn a policy keeping the power grid in security. The possible actions include switching a line status (in service or out-of-service) or changing the line interconnections.
Persodata
Detect Fake paintings
The goal of the challenge is to detect the fake paintings. We present you with real paintings and paintings generated by a computer program. Can you tell them appart? [DIAPOS]
The problem is a binary classification problem. Each sample (image) is characterized by 200 features. You must predict whether the images are fake or real.
The problem is a regression problem (prediction of time of death) with censored data (some people leave the study or are still alive and the end of the study). The metric of evaluation is the concordance index.
Info 232:
Legacy class material from 2018-2019.
Legacy homework from 2018-2019.
Acknowledgements: These challenges are hosted by CodaLab. We received a grant of the FCS Paris-Saclay and EIT Health and sponsorship of Microsoft Azure for Research and Google Research.
Mini Challenges 2017-2018
Challenges
(M2 students)
Abstract
Task
L2 groups
(who solved it)
Computer vision
Autonomous vehicles will become a common means of transportation very soon. However, obstacles remain to be overcome, in particular obstacle avoidance. This requires powerful computer vision algorithms. In this challenge you will contribute to solve the problem of recognizing animals and vehicles.
To illustrate this problematic, we propose to study the image source CIFAR-10 which groups entities that can interact with the vehicle environment like animals(cat, horse, dog, ...) and vehicles (bike, car, truck, ...). We preprocessed the images to you get to solve a multi-class classification problem from pre-computed features. Your score is the balanced accuracy or BAC. It is the average of the error rates for the various classes. Make predictions the are vectors [0 0 ... 1 ... 0 0] with a 1 at the ith position if you want to predict you sample belongs to class i.
Over-prescription of opioids
Over-prescription of opioid medicines presents a new public health problem because many people have become addicted. This challenge asks you to help predicting which doctors tend to over-prescribe such medicines.
The data set contains a binary classification task. The target represents, for each medical prescription whether an opioid has been prescribed or not. The features represent, amongst others, the specialty of the doctor who made the prescription and the name of the non-opioid drugs present in this prescription.
Your score is the Gini or "normalized AUC": 2 AUC - 1. AUC stands for Area under ROC curve. Make numerical predictions for test samples that are larger for the positive class and smaller for the negative class (discriminant values). Random guesses give a score close to 0 while perfect predictions give a score of 1.
House princing
Predicting at which price a house will sell helps people selling their property at a fair price. This dataset contains house sale prices for King County, which includes Seattle. It includes homes sold between May 2014 and May 2015.
This is a regression problem. The dataset contains 19 house features plus the price and the id columns, along with 21613 observations.
Your score is the R-square = 1 - MSE / var(Y).It si 0 for the baseline method that predicts the average target value. It is 1 for perfect guesses. It ca be negative if your predictions are worse than the average target value!
Air quality challenge
Pollution, or the introduction of different forms of waste materials in our environment, has negative effects to the ecosystem we rely on. With modernization and development in our lives, pollution has reached its peak, giving rise to global warming and human illness.
This is a regression problem. The goal of this challenge is to predict the NOx levels in the air in Northern Taiwan, which is an indicator of pollution. The dataset is was initially provided by the Environmental Protection Administration, Executive Yuan, R.O.C.
Your score is the R-square = 1 - MSE / var(Y).It si 0 for the baseline method that predicts the average target value. It is 1 for perfect guesses. It ca be negative if your predictions are worse than the average target value!
Give me some credit
This challenge deals with a fundamental task in the financial industry: credit scoring. In simple English, it means deciding whether to grant a credit to someone or not, depending on her/his historical financial record.
This is a binary classification problem. The data set contains 150000 instances separated on 2 classes, where each class refers to the seriousness of a client in two years.
Your score is the Gini or "normalized AUC": 2 AUC - 1. AUC stands for Area under ROC curve. Make numerical predictions for test samples that are larger for the positive class and smaller for the negative class (discriminant values). Random guesses give a score close to 0 while perfect predictions give a score of 1.
Acknowledgements: These challenges were generated with ChaLab and are hosted by CodaLab. We received a grant of the FCS Paris-Saclay and sponsorship of Microsoft Azure for Research.
Auto-sklearn performances
Mini Challenges 2016-2017
Challenges
(M2 students)
Abstract
Task
L2 groups
(who solved it)
Activity of molecules against HIV
The problem is to relate molecular structure to activity to screen new compounds before actually testing them with High Throughput Screening (HTS) in vitro experiments. HTS is a method for massive scientific experimentation used in drug discovery, linking the fields of biology and chemistry. This method remains very costly process despite many recent technological advances in the field of biotechnology. This is why applying machine learning methods would be of great benefit for the pharmaceutical industry to reduce the number of compounds that need to be tested.
The Objective of is to predict which compounds are active against the AIDS HIV infection. The dataset has two classes : active or inactive (Binary Classification). The variables represent properties of the molecule inferred from its structure.
Note: this project is running on the LRI server. In case of problem, a previous version on the main Codalab instance is available.
Lothlorien
This challenge aims at addressing the issue of resources access (website, drug purchase, violent movie, etc.) based on the age of a person. Indeed a lot of violent content is accessible on the internet and 45 % of children under 12 are not monitored by parental control. For this sake, we rely on the person's real-time image to estimate his age category. Facial aging effects are mainly correlated to bone movement and growth, skin wrinkles and reduction of muscle strength. Human observation lacking of accuracy, we want to find an automatic algorithm to make this distinction.
A computer vision challenge is proposed for undergraduate students in which the challenger must predict the class of a person (major or minor) based on a picture of his/her face.
Note: the main Codalab instance of this challenge has been tested.
Note: this project is running on the LRI server. In case of problem, a previous version on the main Codalab instance is available.
Ecocity
Help SimCity's mayor fight pollution and traffic jams by optimizing the city's bike rental system!
SimCity mayor has invested a lot of money to fight against pollution and reduce traffic jams. Her first action was the purchase of a bike rental system. To improve the system, she wishes to predict the number of bikes rented at each station at any moment of the day using weather data.
The challenge that is to use weather data (temperature, humidity, cloud cover) to predict the number of bikes rented at given station for a given day. To make the challenge more interesting, predictions are asked either in the morning or in the afternoon.
Note: this project is running on the LRI server. In case of problem, a previous version on the main Codalab instance is available.
Movie recommendation
Currently, there are more and more music to listen, movies to watch and things to buy on the Internet. Therefore, developing systems that help users find items they may like is crucial. Recommending items is different from "classical" machine learning, where you only have to predict a class given several features. Recommendation implies using predictions to recommend suitable items (in this case movies) to the adequate people. In addition to that, this preferences can be sometimes evolve in time.
In this challenge, you will work on the famous Movielens dataset. The goal of this challenge is to predict for a user and a given film the score that is the most likely to be awarded by the user.
Note: There is also a LRI version. Warning: both versions were using different score. They should now both use a_metric = 1 - MAE/MAD.
Pick The Sneak Peek
In 2000, 60,234 titles between movies and TV shows were released, according to the IMDB source. In 2010, 165,830 titles and in 2016, 190,275 titles were filmed. We can only notice that the movie release industry is in perpetual increase and the databases aggregating the data are in need of more information to expand.
This is a text processing challenge.
The idea is to facilitate the genre labeling of movies from their summaries and thus to help with categorization of the movies database.
Note: this project is running on the LRI server. In case of problem, a previous version on the main Codalab instance is available.
The Godfather returns!
After last year’s purge accomplished by Batman the Godfather has return and he's looking for new skills, the best criminals in SF, for crime organizations to prosper again and go back to gold age. To make sure about the recruits' abilities, records of their previous crimes in the San Francisco Bay Area are being investigated, background checks are being conducted on the candidates curriculum and a software is being developed to highlight criminals' potential.
The goal is to design software to predict, for each criminal record, the category of crime. If the candidate's crime falls into the category that the Godfather needs, he will be recruited!
Note: No LRI implementation so far.
Acknowledgements: These challenges were generated with ChaLab and are hosted by CodaLab. We received a grant of the FCS Paris-Saclay and sponsorship of Microsoft Azure for Research.
Auto-sklearn performances
Mini Challenges 2015-2016
Challenges
(M2 students)
Abstract
Task
L2 groups
(who solved it)
Diabetes diagnosis
Diabetes will be the seventh most common cause of death in 2030 according to the World Health Organization. In 2014, global prevalence of diabetes was estimated to be more than 9% among adults aged 18+ years. If most hospitals have the necessary medical equipment to treat this disease, some do not have these means.
The task is a binary classification problem. Using the train set, it consists in predicting the length of stay for a patient given its diagnosis and its medications. This label consists in two categories : a stay inferior to 7 days or a stay greater or equal to 7 days.
Restaurant recommendation
We propose a challenge in restaurant recommendation to predict the rating for a particular user of any restaurant. We have very detailed information of the restaurants like geographical information, number of stars, reviews, etc and for each person a list of some restaurants he visited and his personal rate.
The participants will work in two principal tasks:
Task 1: Select the most prevalent features in the three datasets:
Task 2: improving the prediction results using others methods and improving the training dataset with the data of Yelp.
Computer vision
Robots take more place in society everyday and soon they may be walking in the streets among us. There are a lot of problems that need to be solved before that and one of them is adaptation. An AI needs to adapt its vision of the world: when it sees an entity for the first time it should be able to tell if it is a domestic animal, a predator, a vehicle or maybe another robot? That is where transfer learning shows up: extracting general features from specific examples of a group allows to efficiently classify unknown entities.
The idea of the challenge is to learn how to separate distinct classes of images. Precisely, we consider different superclasses, like "aquatic animals", each containing several classes, like "dolphin", and the goal is to tell this superclasses apart.
Crimes in Gotham city
Batman fighting in the forefront to deliver the Gotham City from the evil crimes. And now he and his team want to create a system in order to increase their working efficiency. They have recent years’ crime data of Gotham City which is collected from GCPD and Batman’s database. The data including the location, the time and some other information of each crime. Some crimes have been solved, the others not.
The main goal of this project is to help Batman develop this system. In other words, do the classification of crimes. You can treat it as a binary classification problem, to predict whether a crime can be solved or not. You can also first do the logistic regression to compute how likely a crime will be solved. Then Batman can define the priority for the crimes with this system.
Opinion mining from text
In this project you will tackle the problem of Opinion Mining in movie reviews with a basic set of techniques used in text classification. Many sentiment-analysis methods for the classification of reviews use training and test-data based on star ratings provided by reviewers. However, when reading reviews it appears that the reviewer's ratings do not always give an accurate measure of the sentiment of the review.
The objective of the challenge is to determine the polarity of an opinion from raw text. Since it's a challenge for starter you will only focus on classifying opinion to positive or negative. You can go further in detailing sentiments like happiness, sadness, satisfaction but this will not be our goal in this contest.