CVPRW-NAS 2024 - Day 1 Session 1

Event: CVPR Workshop on Neural Architecture Search 2024 · Duration: 228 min · ▶ Watch on YouTube

Abstract

This video captures the first session of the Fifth Annual CVPR Workshop on Neural Architecture Search (NAS) in 2024. The session begins with a welcome and an introductory overview of NAS, covering its core concepts, challenges, and recent advancements. Subsequent presentations delve into specialized NAS techniques, including zero-shot NAS, differentiable NAS for GANs, quantization-aware NAS for mobile deployment, and connectivity search for convolutional operators. The talks also explore novel proxy metrics for efficient NAS evaluation and the influence of network topology on performance.

Speakers

  • David Towers
  • Radu Marculescu — The University of Texas at Austin
  • Taegun An — Korea University
  • Konstanty Subbotko — University of Warsaw
  • Tianxiao Gao — Ant Group
  • Tunhou Zhang — Duke University
  • Guihong Li — UT Austin/AMD
  • Kartikeya Bhardwaj — CMU/Qualcomm

Talks (10)

  • 00:32:00 — David Towers: Welcome …to the Fifth workshop on Neural Architecture Search (NAS) and Introduction to NAS
    • Introduction to the fifth annual CVPR Workshop on Neural Architecture Search, including the agenda and a basic overview of Neural Architecture Search (NAS).
  • 00:35:54:55 — Radu Marculescu: Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities
    • This talk discusses the challenges, solutions, and opportunities in zero-shot neural architecture search, focusing on hardware-aware NAS and proxy metrics.
  • 00:41:00 — Taegun An: CycleGANAS: Differentiable Neural Architecture Search for CycleGAN
    • This presentation introduces CycleGANAS, a differentiable neural architecture search method specifically designed for CycleGAN, addressing the challenges of complex GAN architectures and limited input data.
  • 00:52:59 — Konstanty Subbotko: The devil is in discretization discrepancy. Robustifying Differentiable NAS with Single-Stage Searching Protocol
    • This talk addresses the discretization error in differentiable NAS by proposing a single-stage searching protocol that eliminates the need for a separate decoding stage, leading to more efficient and robust architecture search.
  • 01:05:45 — Tianxiao Gao: QuantNAS: Quantization-aware Neural Architecture Search For Efficient Deployment On Mobile Device
    • This presentation introduces QuantNAS, a quantization-aware neural architecture search framework designed for efficient deployment of deep learning models on mobile devices, focusing on optimizing both accuracy and hardware-related metrics like latency and energy.
  • 01:23:50 — Tunhou Zhang: CSCO: Connectivity Search of Convolutional Operators
    • This talk presents CSCO, a method for connectivity search of convolutional operators in neural architectures, leveraging graph isomorphism and Metropolis-Hastings Evolutionary Search to efficiently explore large search spaces and find optimal dense connectivity patterns.
  • 01:43:12 — Guihong Li: UP-NAS: Unified Proxy for Neural Architecture Search
    • This presentation introduces UP-NAS, a unified proxy for neural architecture search that combines multiple zero-cost proxies using a learned weighted sum, enabling efficient and competitive performance in various NAS benchmarks.
  • 02:02:29 — Kartikeya Bhardwaj: NN-Mass: How does topology influence gradient propagation and model performance of deep networks with densenet-type skip connections?
    • This talk introduces NN-Mass, a novel metric that quantifies the “mass” of a neural network’s topology, demonstrating its correlation with gradient propagation and model performance, particularly in denseNet-type architectures with skip connections.
  • 02:18:59 — David Towers: Poster Session - Arch 4E (338-355) We will return at 15:50
    • A brief interlude announcing the poster session location and time for returning to the main session.
  • 02:22:00 — David Towers: Closing Remarks
    • Final remarks for the session, including an announcement about an upcoming NAS competition and a thank you to attendees and speakers.

Key Takeaways

  • Zero-shot NAS approaches aim to reduce computational burden by using proxies to estimate architecture quality without full training, often performing well for specific datasets.
  • Discretization error is a significant challenge in differentiable NAS, which can be mitigated by single-stage searching protocols that fine-tune supernets directly, leading to more efficient and robust architecture search.
  • Quantization-aware NAS is crucial for efficient deployment on mobile devices, optimizing for both accuracy and hardware metrics like latency and energy consumption.
  • Novel metrics like NN-Mass and unified proxies (UP-NAS) can effectively quantify network topology and combine multiple zero-cost proxies for improved NAS performance and interpretability.
  • Hardware-aware NAS is an emerging field that considers hardware constraints (e.g., latency, energy) during architecture search, leading to more practical and deployable models, especially for edge devices.

Methods / Models / Datasets Mentioned

  • Multi-layer Perceptron (MLP)
  • GoogleNet
  • ResNet
  • VGG
  • Inception
  • Cutout
  • AutoAugment
  • NAS-Bench-201
  • NAS-Bench-101
  • DARTS
  • ENAS
  • SPOS
  • CycleGANAS
  • CycleGAN
  • AutoGAN
  • AdversarialNAS
  • QuantNAS
  • MobileNetV3
  • ResNet-18
  • BatchQuant
  • ProxylessNAS
  • Once-for-all
  • CSCO
  • Metropolis-Hastings Evolutionary Search (MH-ES)
  • Local Search (LS)
  • Evolutionary Search (ES)
  • UP-NAS
  • NASLib
  • Grad-norm
  • SNIP
  • Synflow
  • Grasp
  • Fisher
  • Jacov
  • Zen-Score
  • NWOT
  • EPE-NAS
  • FLOPS
  • L2-norm
  • Params
  • Plain
  • CIFAR-10
  • DenseNet
  • ImageNet
  • Jacobian Covariant
  • NTK
  • Zen
  • Zico
  • Logdet
  • NN-Mass
  • NN-Degree
  • Variational Graph Isomorphism Autoencoder (VGAE)
  • Tree-structured Parzen Estimator Approach (TPE)
  • ImageNet-1K
  • ImageNet-16-120
  • CIFAR-100
  • Tiny-ImageNet
  • Fashion-MNIST
  • MetaD2A
  • GradSign
  • Graph Convolutional Network (GCN)
  • Densely Connected Neural Architecture Search (DCNAS)
  • DPC (Differentiable Proximal Control)
  • Auto-DeepLab
  • Flash (Fast Neural Architecture Search with Hardware Optimization)

Topics

Neural Architecture Search (NAS) · Zero-shot NAS · Differentiable NAS · Quantization-aware NAS · Hardware-aware NAS · Proxy metrics · Network topology · Generative Adversarial Networks (GANs) · Image-to-image translation · Computational efficiency

Notes

Open for commentary — connections to other work, critiques, follow-up reading.