chan2vec

The methods and experiment results are described in the following paper:

• https://arxiv.org/abs/2010.09892

Directions for running experiments from the paper and generating the https://transparency.tube/ channel classifications data are here.

Directions for doing basic classification with pre-existing embeddings can be found here.

Political Channel Discovery

Commands from the following docs were used for political channel discovery (some data of which is used in Dinkov experiments)

1. Find candidate channels:

• experiments/docs/political_channel_discovery_init_round.txt
• experiments/docs/political_channel_discovery_round1.txt
• experiments/docs/political_channel_discovery_round2.txt
• experiments/docs/political_channel_discovery_round3.txt

2. Final political channel classification, out-of-sample performance stats, and language prediction stats

• experiments/docs/channel_language_prediction.txt
• experiments/docs/political_channel_classification_knn_only.txt

3. Channel discovery hold out analysis

• experiments/docs/political_channel_classification_coverage_analysis.txt

Note, for data collections, all scripts that require --ec2-ip-fp must have a file with the IPs of AWS instances that have been launched. An example can be found here: data_collection/configs/comment_scrape_instances.SAMPLE.txt

Dinkov Media Bias / Fact Check Experiment

The results of comparing chan2vec to Dinkov's model can be generated by running commands in the following doc

• experiments/docs/dinkov_political_preds.txt

Note, the Dinkov predictions were generated by modifying their code so that they would output predictions for individual channels.

Soft Tag Predictions

Experiment results and commands for scoring all out of sample are in this doc:

• experiments/docs/political_soft_tags.txt

Newly Discovered Political Channel Traffic Analysis

Experiment results can be found here:

• experiments/docs/political_soft_tags_traffic_analysis.txt
• experiments/docs/political_soft_tags_traffic_analysis_trends.txt

Transparency.tube Data

The latest data was generated using commands in the following doc:

• experiments/docs/latest_site_data_20201006.txt

Tag definitions can be found here:

• https://github.com/markledwich2/Recfluence

Tag metrics from hold-one-out cross validation:

Political lean metrics from hold-one-out cross validation:

The predictions are available here:

• data/site_preds/labels_20201006/all_political_soft_tags_20201006.txt

Columns are:

• Channel ID
• Probability the channel is political (all over 0.8)
• Soft tag or political lean
• Probability of soft tag or political lean (use threshold of 0.5)

Chan2vec KNN Basic Example

Install numpy and faiss

pip3 install faiss numpy

Download pre-existing emebddings (can reach out to the chan2vec author for these) and add to the specified locations below. Add "--use-gpu True" in order to speed up the command below, otherwise will take > 3 mins.

python3 chan2vec/python/chan2vec_knn.py \
        --vec-fp data/pol_chan_disc/chan2vec_training_data/chan2vec_round3_channels_ds.vectors.txt \
        --chan-info-fp data/pol_chan_disc/chan2vec_training_data/chan2vec_round3_channels_ds.chan_info.txt \
        --label-fp data/datasets/tt_ds_20201031.is_pol.txt \
        --score-chan-fp data/pol_chan_disc/chan2vec_training_data/chan2vec_round3_channels_ds.chan_info.channel_ids.txt \
        --out-fp ./all_political_predictions.txt \
        --num-neighbs 10 --bin-prob True

Get performance of model

python3 chan2vec/python/gen_pred_stats_bin.py \
        --lab-fp data/datasets/tt_ds_20201031.is_pol.txt \
        --score-fp ./all_political_predictions.txt \
        --no-fold-lab-col True --pred-thresh 0.5

Output:

Num instances: 6615
AUC:           0.9906
Accuracy:      0.9587
Precision:     0.8153
Recall:        0.9708

The columns for chan-info-fp are:

• Channel ID
• Assigned int for channel ID
• Channel Name
• Scraped comment subscriptions
• Total subscriptions

The larger the number of "scraped comment subscriptions", the more useful the channel embedding is likely to be for a given task. For political channel classification we filter out all channels with less than 20 "scraped comment subscriptions.