Micro Models Breakthroughs 2023

Anonymous

what were good breakthrough for small and micro models ?

Chat NeurIPS logo
Chat NeurIPS

The field of small and micro models has seen several innovative breakthroughs, focusing on improving computational efficiency, reducing model size, and maintaining high performance. Here are some notable advancements:

  1. Leveraging Early-Stage Robustness in Diffusion Models: This study presents a method to speed up noise estimation in diffusion models by exploiting the robustness of early-stage diffusion models. It introduces low-bit activation for early reverse diffusion while maintaining high-bit activation for later stages, achieving faster computation without sacrificing image quality【Kim, Y., Jo, D., Jeon, H., Kim, T., Ahn, D., Kim, H., & kim, j.-j. (2023). Leveraging Early-Stage Robustness in Diffusion Models for Efficient and High-Quality Image Synthesis Poster. Page 2】.

  2. CORNN for Rapid Inference of Neural Dynamics: The CORNN model offers a new approach to bridge the gap between artificial networks and hand-crafted network models, especially for recordings of small numbers of cells. It retains the advantages of neural networks while overcoming traditional limitations【Dinc, F., Shai, A., Schnitzer, M., & Tanaka, H. (2023). CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics Poster. Page 2】.

  3. Hardware-Oriented Transformers for BEV Perception: The research specifically targets small models in BEV perception, demonstrating favorable results for these models compared to other studies. It provides a detailed analysis of model configurations and their impact on temporal modeling and robustness【Dong, P., Kong, Z., Meng, X., Yu, P., Gong, Y., Yuan, G., Tang, H., & Wang, Y. (2023). HotBEV: Hardware-oriented Transformer-based Multi-View 3D Detector for BEV Perception Poster. Page 8】.

  4. MCUFormer for Vision Transformers on Microcontrollers: This study introduces MCUFormer, a method for deploying vision transformers on microcontrollers with limited memory. It involves hardware-algorithm co-optimizations, including a one-shot network architecture search for memory-efficient design, and customizing the inference operator library to fit the memory constraint【Liang, Y., Wang, Z., Xu, X., Tang, Y., Zhou, J., & Lu, J. (2023). MCUFormer: Deploying Vision Tranformers on Microcontrollers with Limited Memory Poster. Page 3】.

  5. Diffusion Model Adaptation for Text-to-Image Synthesis: This research focuses on adapting text-to-image diffusion models to handle a variety of sizes and aspect ratios while maintaining visual fidelity. It introduces a scaling factor to alleviate changes in attention entropy, enabling better visual effects and image quality without additional training【Jin, Z., Shen, X., Li, B., & Xue, X. (2023). Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis Poster. Page 1】.

  6. Channel-Adaptive Models in Microscopy Imaging: The study investigates channel-adaptive models for microscopy imaging, finding that they can generalize better to out-of-domain tasks and are computationally efficient. It includes extensive experiments with different methods and a dataset of varied-channel single-cell images【Chen, Z. S., Pham, C., Wang, S., Doron, M., Moshkov, N., Plummer, B., & Caicedo, J. C. (2023). CHAMMI: A benchmark for channel-adaptive models in microscopy imaging Poster. Page 10】.

  7. Machine Learning Force Fields for Semiconductor Simulations: This paper presents a benchmark suite for evaluating machine learning force fields in semiconductor simulations. It aims to offer a less costly alternative to traditional methods, especially for large-scale atomic-level simulations【Kim, G., Na, B., Kim, G., Cho, H., Kang, S., Lee, H. S., Choi, S., Kim, H., Lee, S., & Kim, Y. (2023). Benchmark of Machine Learning Force Fields for Semiconductor Simulations: Datasets, Metrics, and Comparative Analysis Poster. Page 1】.

These breakthroughs demonstrate the potential