Designing Concise ConvNets with Columnar Stages

2024-10-05 · Ashish Kumar, Jaesik Park

In the era of vision Transformers, the recent success of VanillaNet shows the huge potential of simple and concise convolutional neural networks (ConvNets). Where such models mainly focus on runtime, it is also crucial to simultaneously focus on other aspects, e.g., FLOPs, parameters, etc, to strengthen their utility further. To this end, we introduce a refreshing ConvNet macro design called Columnar Stage Network (CoSNet). CoSNet has a systematically developed simple and concise structure, smaller depth, low parameter count, low FLOPs, and attention-less operations, well suited for resource-constrained deployment. The key novelty of CoSNet is deploying parallel convolutions with fewer kernels fed by input replication, using columnar stacking of these convolutions, and minimizing the use of 1x1 convolution layers. Our comprehensive evaluations show that CoSNet rivals many renowned ConvNets and Transformer designs under resource-constrained scenarios. Code: https://github.com/ashishkumar822/CoSNet

📄 PDF Abstract BibTeX arXiv:2410.04089

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Adam 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Hierarchical internal representation of spectral features in deep convolutional networks trained for EEG decoding

2017-11-21 · Kay Gregor Hartmann, Robin Tibor Schirrmeister, Tonio Ball

Recently, there is increasing interest and research on the interpretability of machine learning models, for example how they transform and internally represent EEG signals in Brain-Computer Interface (BCI) applications. …

Brain Computer Interface EEG Eeg Decoding Electroencephalogram (EEG)+2

Columnar Database Techniques for Creating AI Features

2017-12-07 · Brad Carlile, Akiko Marti, Guy Delamarter

Recent advances with in-memory columnar database techniques have increased the performance of analytical queries on very large databases and data warehouses. At the same time, advances in artificial intelligence (AI) alg…

feature selection

ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases

2017-11-30 · ECCV 2018 9 · Pierre Stock, Moustapha Cisse

ConvNets and Imagenet have driven the recent success of deep learning for image classification. However, the marked slowdown in performance improvement combined with the lack of robustness of neural networks to adversari…

image-classification Image Classification

Scaling Up Your Kernels: Large Kernel Design in ConvNets towards Universal Representations

2024-10-10 · Yiyuan Zhang, Xiaohan Ding, Xiangyu Yue

This paper proposes the paradigm of large convolutional kernels in designing modern Convolutional Neural Networks (ConvNets). We establish that employing a few large kernels, instead of stacking multiple smaller ones, ca…

Time Series Forecasting Video Recognition

UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio Video Point Cloud Time-Series and Image Recognition

2024-01-01 · CVPR 2024 1 · Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao 외

Large-kernel convolutional neural networks (ConvNets) have recently received extensive research attention but two unresolved and critical issues demand further investigation. 1) The architectures of existing large-ke…

Time Series Time Series Forecasting