Portada de A Supervised Learning Framework in the Context of Multiple Annotators
Acceso abierto E-book

A Supervised Learning Framework in the Context of Multiple Annotators

Walter Julián Gil González

Editorial UNAL - Universidad Nacional de Colombia

Sinopsis

The increasing popularity of crowdsourcing platforms, i.e., Amazon Mechanical Turk, is changing how datasets for supervised learning are built. In these cases, instead of having datasets labeled by one source (which is supposed to be an expert who provided the absolute gold standard), we have datasets labeled by multiple annotators with different and unknown expertise. Hence, we face a multi-labeler scenario, which typical supervised learning models cannot tackle.For this reason, much attention has recently been given to the approaches that capture multiple annotators’ wisdom. 

Contenido

1 Preliminaries 2

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.3 Mathematical Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3.1 Methods for Supervised Learning . . . . . . . . . . . . . . . . . . . . . 5

1.3.2 Learning from Multiple Annotators . . . . . . . . . . . . . . . . . . . . 10

1.4 Literature Review on Supervised Learning from Multiple Annotators . . . . . . . 10

1.5 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.5.1 General Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.5.2 Specific Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.6 Outline and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

1.6.1 Kernel Alignment-Based Annotator Relevance Analysis (KAAR) . . . . . . 17

1.6.2 Localized Kernel Alignment-Based Annotator Relevance

Analysis (LKAAR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1.6.3 Regularized Chained Deep Neural Network for Multiple Annotators (RCDNN) 19

1.6.4 Chained Gaussian Processes for Multiple Annotators (CGPMA) and

Correlated Chained Gaussian Processes for Multiple Annotators

(CCGPMA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

1.6.5 Book Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2 Kernel Alignment-Based Annotator Relevance Analysis 24

2.1 Centered Kernel Alignment Fundamentals . . . . . . . . . . . . . . . . . . . . . 24

2.2 Kernel Alignment-Based Annotator Relevance Analysis . . . . . . . . . . . . . . 25

2.2.1 KAAR for Classification and Regression . . . . . . . . . . . . . . . . . . 27

2.3 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.3.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.3.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.4.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.4.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39


3 Localized Kernel Alignment-Based Annotator Relevance Analysis 40

3.1 Localized Kernel Alignment Fundamentals . . . . . . . . . . . . . . . . . . . . 40

3.2 Localized Kernel Alignment-Based Annotator Relevance Analysis . . . . . . . . . 41

3.2.1 LKAAR for Classification and Regression . . . . . . . . . . . . . . . . . 43

3.3 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.3.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.3.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.4.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.4.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4 Regularized Chained Deep Neural Network for Multiple Annotators 59

4.1 Chained Deep Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4.2 Regularized Chained Deep Neural Network for Classification with Multiple Annotators 60

4.3 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.3.1 Tested Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

4.3.2 Provided and Simulated Annotations . . . . . . . . . . . . . . . . . . . 62

4.3.3 Method Comparison and Quality Assessment . . . . . . . . . . . . . . . 62

4.3.4 RCDNN Detailed Architecture and Training . . . . . . . . . . . . . . . . 63

4.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

5 Correlated Chained Gaussian Processes for Multiple Annotators 73

5.1 Chained Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

5.1.1 Correlated Chained Gaussian Processes . . . . . . . . . . . . . . . . . 75

5.2 Correlated Chained GP for Multiple Annotators-CCGPMA . . . . . . . . . . . . . 77

5.2.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.2.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

5.3 Experimental Set-Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5.3.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

5.3.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

5.4 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

5.4.1 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

5.4.2 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

6 Final Remarks 95

6.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95


6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

6.3 Repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Bibliography 101

Appendices 106

Appendix A CCGPMA Supplementary Material 107

A.1 Derivation of CCGPMA Lower Bounds . . . . . . . . . . . . . . . . . . . . . . . 107

A.1.1 Gradients w.r.t. the Variational Parameters . . . . . . . . . . . . . . . . 108

A.2 Likelihood Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

A.2.1 Multiclass Classification with Multiple Annotators . . . . . . . . . . . . 109

A.2.2 Gaussian Distribution for Regression with Multiple Annotators . . . . . . 114

Alphabetical Index 118

Materias

BISAC

  • COM014000. ORDENADORES > Ciencias de la Computación

Thema

  • UT. Redes y comunicaciones informáticas

Dewey

  • 621.3. Tecnología (ciencias aplicadas) > Ingeniería y operaciones afines > Física Aplicada > Eléctrico, electrónico, magnético, comunicaciones, ingeniería informática; iluminación

Propias de la editorial

  • Ingeniería
  • Ingeniería de sistemas

Más de Editorial UNAL - Universidad Nacional de Colombia

Ver todas