Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfaction

2025-02-03

AI Summary

This work aims to enhance the performance of the fruit-fly-inspired neural network FlyVec on word/sentence embedding tasks—specifically, whether its capabilities can surpass the original model and approach those of large state-of-the-art (SOTA) models without increasing parameter count. To this end, we introduce complex-valued weights into FlyVec’s sparse coding framework for the first time, leveraging phase information to implicitly encode positional structure, thereby constructing a single-layer unsupervised sequence encoder. This enables a leap from static word embeddings to context-aware sentence representations with zero additional parameters. The method preserves biological interpretability and computational efficiency, yielding sparse, interpretable sentence representations. It significantly outperforms FlyVec across multiple semantic similarity and downstream NLP benchmarks, matching the performance of SOTA models orders of magnitude larger in parameter count. Our core contribution is a novel complex-domain sparse projection mechanism, establishing a new paradigm for lightweight neuro-symbolic hybrid modeling.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Biologically inspired neural networks offer alternative avenues to model data distributions. FlyVec is a recent example that draws inspiration from the fruit fly's olfactory circuit to tackle the task of learning word embeddings. Surprisingly, this model performs competitively even against deep learning approaches specifically designed to encode text, and it does so with the highest degree of computational efficiency. We pose the question of whether this performance can be improved further. For this, we introduce Comply. By incorporating positional information through complex weights, we enable a single-layer neural network to learn sequence representations. Our experiments show that Comply not only supersedes FlyVec but also performs on par with significantly larger state-of-the-art models. We achieve this without additional parameters. Comply yields sparse contextual representations of sentences that can be interpreted explicitly from the neuron weights.

Problem

• Improve FlyVec model efficiency
• Learn sequence representations effectively
• Achieve competitive performance with simplicity

Innovation

• Complex weights for sequences
• Single-layer neural network
• Sparse contextual representations