Morph Call

홈페이지 · 논문 2편

Morph Call is a suite of 46 probing tasks for four Indo-European languages that fall under different morphology: Russian, French, English, and German. The tasks are designed to explore the morphosyntactic content of multilingual transformers which is a less studied aspect at the moment. The tasks are divided into four groups: * Morphosyntactic Features: probe the encoder for the occurrence of the morphosyntactic properties. * Masked Token: analogous to Morphosyntactic Features with the exception that the target word is replaced with a tokenizer-specific mask token. * Morphosyntactic Values: is a group of k-way classification tasks for each feature where *k* is the number of values that the feature can take. * Perturbations: tasks test the encoder sensitivity to syntactic and inflectional sentence perturbations. ## Probing Methods * Supervised probing involves training a Logistic Regression classifier to predict a property. The performance is used as a proxy to evaluate the model knowledge. * Neuron-level Analysis [Durrani et al., 2020] allows retrieving a group of individual neurons that are most relevant to predict a linguistic property. * Contextual Correlation Analysis [Wu et al., 2020] is a representation-level similarity measure that allows identifying pairs of layers of similar behavior. ## Usage We provide an example of the experiment on Masked Token task (Case, German). `` bash me@my-laptop:~$ python3 probe.py --help INFO: Showing help with the command 'probe.py -- --help'. NAME probe.py - configure the experiment AND perform probing SYNOPSIS probe.py <flags> DESCRIPTION configure the experiment AND perform probing FLAGS --results_path=RESULTS_PATH Type: Optional[str] Default: None path to a folder to store the probing results and the model intermediate activations --model_architecture=MODEL_ARCHITECTURE Type: typ... Default: 'bert multilingual' --model_is_finetuned=MODEL_IS_FINETUNED Type: bool Default: False if to perform the experiment on the fine-tuned model --model_finetuned_path=MODEL_FINETUNED_PATH Type: Optional[str] Default: None (only if model_is_finetuned is True) path to store the fine-tuned model --model_finetuned_config_google_url=MODEL_FINETUNED_CONFIG_GOOGLE_URL Type: Optional[] Default: None (only if model_is_finetuned is True) the url of the fine-tuned model config if to be downloaded --model_finetuned_model_google_url=MODEL_FINETUNED_MODEL_GOOGLE_URL Type: Optional[] Default: None (only if model_is_finetuned is True) the url of the fine-tuned model weights if to be downloaded --model_is_random=MODEL_IS_RANDOM Type: bool Default: False if to perform the random initialization of the model --layers_to_probe=LAYERS_TO_PROBE Type: List Default: 'all' (either "all" or list w. possible numbers from 0 to 11) -- model layers to probe. e.g.: [1, 3, 11], or "all" --train_n_sentences=TRAIN_N_SENTENCES Type: int Default: 1500 number of sentences used to train the probing classifier --test_n_sentences=TEST_N_SENTENCES Type: int Default: 1000 number of sentences used to evaluate the probing classifier --dev_n_sentences=DEV_N_SENTENCES Type: int Default: 0 DEPRECATED ``