THESIS

Implementation of Dimensionality Reduction on Word Embeddings Vector Generated

by Bidirectional Encoder Representation from Transformers (BERT)

ABSTRACT

This research focuses on improving the efficiency of complex artificial intelligence models, such as BERT, by applying dimension reduction techniques. The BERT model has millions of parameters, resulting in high computational and memory requirements during training. The approach taken utilizes BERT as a foundation applying whitening or sphering techniques as a dimension reduction method at the feature extraction stage. Two scenarios are evaluated: a standard benchmark using the original BERT and a modified scenario involving BERT feature extraction, whitening techniques (PCA, ZCA, BERT Whitening), and classification using Bi-LSTM or MLP. The AG News dataset, containing news headlines and descriptions with four topic classes. is the main focus of the research. Results show that the model in the modified scenario, which combines BERT features, J. Su whitening, and a Bi-LSTM classifier, provides the best performance in terms of accuracy, F1 score, training time, and Graphics Processing Unit (GPU) memory usage. These findings indicate that whitening dimension reduction can improve text classification efficiency without sacrificing accuracy. This research is expected to expand AI applications in resource- constrained environments by improving the efficiency of complex models like BERT through parameter optimization and dimension reduction.

PUBLICATION

• LC-BERT: Dimensionality Reduction on BERT’s Vector Embeddings (link)