Optimizer Plugger and NoiseLayer: Elegant way of Deep Learning


Creative Commons License

Ertuğrul M. S., Yapalı R., Taşdelen A., Konuralp A.

The 4th International Conference on Applied Mathematics in Engineering (ICAME’26), Balıkesir, Türkiye, 2 - 04 Eylül 2026, cilt.1, ss.57, (Özet Bildiri)

  • Yayın Türü: Bildiri / Özet Bildiri
  • Cilt numarası: 1
  • Basıldığı Şehir: Balıkesir
  • Basıldığı Ülke: Türkiye
  • Sayfa Sayıları: ss.57
  • Açık Arşiv Koleksiyonu: AVESİS Açık Erişim Koleksiyonu
  • Manisa Celal Bayar Üniversitesi Adresli: Evet

Özet

This study leverages the performance of deep learning through adaptive, drop-in algorithms. We propose two novel
methods, namely NoiseLayer and Optimizer Plugger, designed as seamless replacements for standard literature
components. The NoiseLayer calculates the noise according to the Gaussian noise and scales it with the entropy
of the input, so the noise is adaptive. The adaptive noise feature fits various domains, from computer vision to
natural language processing (NLP). The current noise calculations mostly require a sensitive tunable parameter
that leads us to fewer outcomes. This problem can be seen in language tasks perfectly. Also, the noise algorithms
vary according to their purposes, and this variation led us to the long tuning sessions. The tuning sessions drain
the hardware resources of the researchers for decades. However, the proposed NoiseLayer solves various algorithm
phase testing entirely with a drop-in replacement feature of the algorithm. The other adaptive algorithm is the
optimizer plugger; it creates its own hyperparameters via evolutionary properties. The initialized parameters for
each hyperparameter create a population, and the optimizer plugger chooses the best fit through steps. Then the
population updates itself according to the fitness function i.e. loss function of the training. The optimizer plugger
has also a drop-in replacement feature, and it has been demonstrated via testing different optimization algorithms.
The main purpose of the optimizer plugger is not to outperform the existing algorithms, its purpose is to reduce
the hyperparameter sensitivity of the attached optimization algorithm while maintaining the performance of the
attached algorithm. The experiments and benchmarks of the proposed algorithms demonstrate how researchers can
use the suggested methods. These experiments and benchmarks prove that deep learning can benefit from adaptive
and drop-in solutions.