Graph representation learning

Learning text representations with inductive GNNs

How do graph architectures change the way individual documents are represented and classified?

Saran Pandian, Uttkarsh Chaurasia, Shudhanshu Ranjan, Shefali Saxena

Conference paper · IEEE ASSIC 2022 · 2022 · Co-author

Read the paper Author preprint Explore results

3graph architectures

3text benchmarks

98.1%DBpedia accuracy · GAT

Published conference paper: IEEE ASSIC 2022. An author preprint is also available through EasyChair.

My contribution. I conducted this work during my research role at LBR Research Labs, comparing TexTING, inductive GCN, and inductive GAT for document classification, and co-authored the IEEE ASSIC 2022 paper.

A graph showing one-hop and two-hop node neighborhoods, alongside a pipeline comparing gated recurrent, graph convolutional, and graph attention models with a shared readout layer. — Our document-graph pipeline: word co-occurrence and feature matrices feed one of three graph encoders, followed by a shared readout layer.Figure 1, Pandian, Chaurasia, Ranjan & Saxena · IEEE ASSIC 2022 · DOI: 10.1109/ASSIC55218.2022.10088315

01 / THE QUESTION

Background.

Corpus-wide text graphs couple document representation to a graph of the entire collection. Inductive approaches instead build a separate graph for each document, making document-level representation less dependent on the complete corpus graph.

Building on TexTING, this study compares gated graph recurrent units, graph convolution, and graph attention under a common document-graph pipeline. It also examines message-passing depth and learning behavior across datasets of different sizes.

02 / THE APPROACH

Method, step by step.

Open a step to explore the approach.

01Construct a document graph
Each unique word is a node. Word co-occurrence within a window of size three determines adjacency, and 300-dimensional GloVe vectors initialize node features.
02Compare graph encoders
The study evaluates the TexTING gated graph recurrent model, an inductive graph convolutional network, and an inductive graph attention network. Attention learns the relative importance of neighboring nodes.
03Read out a document representation
The shared readout uses learned feature transformations, mean pooling, and max pooling to create a document vector, followed by softmax classification.
04Evaluate accuracy and training behavior
We compared classification accuracy and cross-entropy on AG News, IMDB, and DBpedia, examining model architecture, dataset size, and one-to-four-hop message passing.

Study design

We tested on AG News (news classification), IMDB (sentiment classification), and DBpedia (ontology classification). Models used GloVe word features, a co-occurrence window of three, Glorot initialization, and the Adam optimizer. Accuracy is shown as a percentage.

03 / THE EVIDENCE

Results.

Inductive GAT achieved the highest accuracy of the three models on every benchmark. Select a dataset to compare the results.

AG News · four-class news classification

Accuracy from Table II. Inductive GAT improves on TexTING by 0.3 percentage points.

Accuracy · scale 0–100%

TexTING 91.1%

Inductive GCN 91.2%

Inductive GAT 91.4%

0 100

AG News · four-class news classification
Model	Accuracy
TexTING	91.1%
Inductive GCN	91.2%
Inductive GAT	91.4%

IMDB · binary sentiment classification

Accuracy from Table II. Inductive GAT improves on TexTING by 1.1 percentage points.

Accuracy · scale 0–100%

TexTING 86.7%

Inductive GCN 86.6%

Inductive GAT 87.8%

0 100

IMDB · binary sentiment classification
Model	Accuracy
TexTING	86.7%
Inductive GCN	86.6%
Inductive GAT	87.8%

DBpedia · fourteen-class ontology classification

Accuracy from Table II. Inductive GAT improves on TexTING by 1.4 percentage points.

Accuracy · scale 0–100%

TexTING 96.7%

Inductive GCN 98.0%

Inductive GAT 98.1%

0 100

DBpedia · fourteen-class ontology classification
Model	Accuracy
TexTING	96.7%
Inductive GCN	98.0%
Inductive GAT	98.1%

Numerical results: Pandian et al., IEEE ASSIC 2022, Table II.

04 / WHAT IT MEANS

Conclusions.

Learned attention gave the best accuracy among the three document-level graph architectures we tested.
We observed different early training cross-entropy behavior on IMDB and DBpedia as model complexity and dataset size changed.
Global attention and comparisons with transformer models are directions for further study.

Limitations & next steps

Our experiments compare three graph architectures across three datasets. Transformer baselines and formal uncertainty-calibration evaluation would extend this comparison.

05 / READ FURTHER

Citation & sources.

Comparative Study of Inductive Graph Neural Network Models for Text Classification

IEEE ASSIC 2022 · International Conference on Advancements in Smart, Secure and Intelligent Computing · pp. 227–231

Citation

Pandian S, Chaurasia U, Ranjan S, Saxena S. Comparative Study of Inductive Graph Neural Network Models for Text Classification. 2022 International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC). 2022:227–231. https://doi.org/10.1109/ASSIC55218.2022.10088315

Publisher record Author preprint

Published conference paper: IEEE ASSIC 2022. An author preprint is also available through EasyChair.

Explore all research