Institution profile

German University in Cairo

Academic institutionafrica · eg
Official website
Research library26linked papers
Opportunities0open roles
Selected work

Representative Papers

RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers

Aug 15, 2026

This study addresses the loss of representational diversity caused by deep weight sharing in Transformers by proposing RecurrentGPT. The method employs fixed initial and final modules encapsulating an iterable core, incorporating a gating recurrent modulation mechanism based on hidden states and noise to enable functional specialization within a few shared layers during iteration. This approach overcomes the limitations of traditional parameter reuse. Experiments demonstrate that a three-layer model matches the accuracy of a 12-layer GPT-2 Small. At larger scales, RecurrentGPT reduces parameters by 63% and peak memory usage by 59% while achieving significantly lower validation loss than non-recurrent baselines, effectively balancing expressive capacity with memory efficiency.

0 citationsRead paper
Recent publications

Latest Papers

RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers

Aug 15, 2026

This study addresses the loss of representational diversity caused by deep weight sharing in Transformers by proposing RecurrentGPT. The method employs fixed initial and final modules encapsulating an iterable core, incorporating a gating recurrent modulation mechanism based on hidden states and noise to enable functional specialization within a few shared layers during iteration. This approach overcomes the limitations of traditional parameter reuse. Experiments demonstrate that a three-layer model matches the accuracy of a 12-layer GPT-2 Small. At larger scales, RecurrentGPT reduces parameters by 63% and peak memory usage by 59% while achieving significantly lower validation loss than non-recurrent baselines, effectively balancing expressive capacity with memory efficiency.

0 citationsRead paper