Our data sets

Examples

We use four data sets throughout the course.

MNIST, handwritten digits

70000 images of handwritten digits, 28 by 28 pixels, with a label from 0 to 9.

import warnings
import numpy as np
import pandas as pd

warnings.filterwarnings("ignore")
from sklearn.datasets import fetch_openml

mnist = fetch_openml("mnist_784", version=1, as_frame=True, parser="auto")
X, y = mnist.data, mnist.target

print("images:", X.shape)
print("labels:", y.shape)
print("classes:", " ".join(sorted(y.unique().tolist())))

1

The download is about 15 MB and is cached, so this is slow only the first time. The original data set is on OpenML with id 554.

images: (70000, 784)
labels: (70000,)
classes: 0 1 2 3 4 5 6 7 8 9

Every row is one image, flattened into 784 numbers between 0 and 255. To look at one we fold it back into 28 by 28.

import matplotlib.pyplot as plt

i = 18
fig, ax = plt.subplots(figsize=(2.4, 2.4))
ax.imshow(X.iloc[i].values.reshape(28, 28) / 255, cmap="gray")
ax.set(xticks=[], yticks=[])
ax.set_title(f"label: {y[i]}")
plt.show()

Figure 4.1: Row 18 of the data set, and the label that goes with it.

fig, axes = plt.subplots(1, 10, figsize=(6.6, 0.9))
for d, ax in enumerate(axes):
    j = int(np.flatnonzero(y == str(d))[0])
    ax.imshow(X.iloc[j].values.reshape(28, 28) / 255, cmap="gray")
    ax.set(xticks=[], yticks=[])
    ax.set_title(str(d), fontsize=8)
plt.show()

Figure 4.2: The first example of each digit. Some are written in ways that are hard even for a person.

Spam

A set of emails labelled as spam or ham. The input is text, so we will need a representation before we can fit anything.

spam = pd.read_csv("https://go.epfl.ch/bio322-spam.csv")

print("emails:", len(spam))
spam["label"].value_counts()

emails: 11512

label
spam    6000
ham     5512
Name: count, dtype: int64

The text column is long, so here are the first hundred characters of a few of each.

spam[spam["label"] == "spam"].head(10)

	label	text
1	spam	re swollen ear i final a mor t gag e l o an th...
3	spam	get it up again hello tri this revolutionari p...
8	spam	re for you hello again i sent you an email a f...
9	spam	get great qualiti pictur with a free megapixel...
11	spam	window xp pro with servic pack now onli
12	spam	co gone crazi click here to be remov
16	spam	discount microsoft money standard get softwar ...
17	spam	new peniss enlarg patch inroad good morn sir w...
21	spam	some nasti daughter are forc to fuck her dad c...
22	spam	b uy pain medicin onlin look for vlcodln and o...

spam[spam["label"] == "ham"].head(10)

	label	text
0	ham	start date hourahead hour start date hourahead...
2	ham	re ena associ analyst kitchen louis cc hilli k...
4	ham	hi hello there mister bill just a quick note t...
5	ham	kate s birthday parti i will be year old this ...
6	ham	mark haedick vacat mark haedick will be on vac...
7	ham	fw power gtc louis carl and i have done a quic...
10	ham	hous warm bash staci s parti staci s saturday ...
13	ham	schedul c reserv we have had troubl valu trans...
14	ham	california litig team start on thursday april ...
15	ham	celebr respons over the last coupl of year we ...

Note that there is nothing numeric here at all. Turning these emails into a table of numbers is a choice we have to make, and we come back to it in week 3.

Weather

Hourly measurements from Swiss weather stations. We want to predict the wind peak in Luzern five hours ahead.

import numpy as np
import pandas as pd

weather = pd.read_csv("https://go.epfl.ch/bio322-weather2015-2018.csv")
weather.head(10)

	time	BAS_wind_peak	BAS_pressure	BAS_temperature	BAS_precipitation	BAS_sunshine_duration	BAS_wind_mean	BAS_wind_direction	GVE_wind_peak	GVE_pressure	...	BER_sunshine_duration	BER_wind_mean	BER_wind_direction	CHU_wind_peak	CHU_pressure	CHU_temperature	CHU_precipitation	CHU_sunshine_duration	CHU_wind_mean	CHU_wind_direction
0	2015010100	8.6	997.1	-5.2	0.0	0.0	4.3	198	26.3	984.3	...	0.0	13.3	24	24.5	968.3	-3.1	0.0	0.0	7.9	194
1	2015010101	6.5	997.3	-6.1	0.0	0.0	3.2	201	13.0	984.9	...	0.0	9.7	25	16.9	968.7	-4.2	0.0	0.0	4.7	207
2	2015010102	5.0	997.6	-5.6	0.0	0.0	2.2	146	9.4	985.3	...	0.0	8.6	19	24.5	968.7	-4.4	0.0	0.0	7.6	204
3	2015010103	5.8	997.6	-6.1	0.0	0.0	2.5	143	17.3	985.4	...	0.0	1.8	90	10.4	968.5	-4.9	0.0	0.0	4.3	221
4	2015010104	6.5	997.7	-6.1	0.0	0.0	3.6	198	8.6	985.3	...	0.0	1.8	117	16.6	968.3	-4.7	0.0	0.0	5.4	205
5	2015010105	6.8	997.9	-6.6	0.0	0.0	3.2	172	7.9	985.5	...	0.0	2.2	135	10.8	968.5	-4.9	0.0	0.0	3.2	220
6	2015010106	15.5	997.7	-5.0	0.0	0.0	6.1	120	8.6	985.8	...	0.0	1.8	143	11.2	969.2	-5.8	0.0	0.0	3.2	194
7	2015010107	17.6	997.9	-4.1	0.0	0.0	6.8	93	6.5	986.0	...	0.0	1.1	0	7.2	969.4	-6.3	0.0	0.0	2.9	219
8	2015010108	6.5	998.4	-5.2	0.0	18.0	2.2	194	11.2	986.7	...	15.0	2.2	166	10.1	969.6	-5.6	0.0	0.0	5.0	194
9	2015010109	11.9	999.0	-4.2	0.0	9.0	3.2	164	11.5	986.9	...	60.0	1.4	194	14.8	969.8	-4.7	0.0	35.0	6.8	183

10 rows × 57 columns

Bicycle rentals

The number of bicycles rented in Washington DC in a given hour, together with the weather and the time. It is on OpenML with id 42712.

from sklearn.datasets import fetch_openml

bikes = fetch_openml(data_id=42712, as_frame=True, parser="auto")
print(bikes.DESCR)

Bike sharing systems are new generation of traditional bike rentals where whole process from membership, rental and return 
back has become automatic. Through these systems, user is able to easily rent a bike from a particular position and return 
back at another position. Currently, there are about over 500 bike-sharing programs around the world which is composed of 
over 500 thousands bicycles. Today, there exists great interest in these systems due to their important role in traffic, 
environmental and health issues. 

Apart from interesting real world applications of bike sharing systems, the characteristics of data being generated by
these systems make them attractive for the research. Opposed to other transport services such as bus or subway, the duration
of travel, departure and arrival position is explicitly recorded in these systems. This feature turns bike sharing system into
a virtual sensor network that can be used for sensing mobility in the city. Hence, it is expected that most of important
events in the city could be detected via monitoring these data.

Bike-sharing rental process is highly correlated to the environmental and seasonal settings. For instance, weather conditions,
precipitation, day of week, season, hour of the day, etc. can affect the rental behaviors. The core data set is related to  
the two-year historical log corresponding to years 2011 and 2012 from Capital Bikeshare system, Washington D.C., USA which is 
publicly available in http://capitalbikeshare.com/system-data. We aggregated the data on two hourly and daily basis and then 
extracted and added the corresponding weather and seasonal information. Weather information are extracted from http://www.freemeteo.com. 

Use of this dataset in publications must be cited to the following publication:
Fanaee-T, Hadi, and Gama, Joao, "Event labeling combining ensemble detectors and background knowledge", 
Progress in Artificial Intelligence (2013): pp. 1-15, Springer Berlin Heidelberg, doi:10.1007/s13748-013-0040-3.

Attributes:
- season : season (1:springer, 2:summer, 3:fall, 4:winter)
- yr : year (0: 2011, 1:2012)
- mnth : month ( 1 to 12)
- hr : hour (0 to 23)
- holiday : weather day is holiday or not (extracted from http://dchr.dc.gov/page/holiday-schedule)
- weekday : day of the week
- workingday : if day is neither weekend nor holiday is 1, otherwise is 0.
- weathersit : 
    - 1: Clear, Few clouds, Partly cloudy, Partly cloudy
    - 2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist
    - 3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds
    - 4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog
- temp : Normalized temperature in Celsius. The values are divided to 41 (max)
- atemp: Normalized feeling temperature in Celsius. The values are divided to 50 (max)
- hum: Normalized humidity. The values are divided to 100 (max)
- windspeed: Normalized wind speed. The values are divided to 67 (max)
- casual: count of casual users
- registered: count of registered users
- cnt: count of total rental bikes including both casual and registered

This version was cleanup up by Joaquin Vanschoren:
- Category labels replaced by category names (season, weathersit, year)
- Turned back normalization for temperature and windspeed for interpretability
- Renamed features for readability

Downloaded from openml.org.

bikes.frame.head(10)

	season	year	month	hour	holiday	weekday	workingday	weather	temp	feel_temp	humidity	windspeed	count
0	spring	0	1	0	False	6	False	clear	9.84	14.395	0.81	0.0000	16
1	spring	0	1	1	False	6	False	clear	9.02	13.635	0.80	0.0000	40
2	spring	0	1	2	False	6	False	clear	9.02	13.635	0.80	0.0000	32
3	spring	0	1	3	False	6	False	clear	9.84	14.395	0.75	0.0000	13
4	spring	0	1	4	False	6	False	clear	9.84	14.395	0.75	0.0000	1
5	spring	0	1	5	False	6	False	misty	9.84	12.880	0.75	6.0032	1
6	spring	0	1	6	False	6	False	clear	9.02	13.635	0.80	0.0000	2
7	spring	0	1	7	False	6	False	clear	8.20	12.880	0.86	0.0000	3
8	spring	0	1	8	False	6	False	clear	9.84	14.395	0.75	0.0000	8
9	spring	0	1	9	False	6	False	clear	13.12	17.425	0.76	0.0000	14

The response is a count.