Multimodal: Computer Vision or Audio/Speech, Student's Choice

About the course

This course deepens specialization in one of two multimodal tracks: Computer Vision (detection, segmentation, vision-language models) or Audio/Speech (ASR, TTS, speaker diarization, multimodal audio models). It covers both modality-specific architectural choices and general multimodal fusion approaches. The course prepares students for the research track (s3-ai-research) and applied projects (s3-applied-ai).

What you'll learn

✓Build and fine-tune a specialized multimodal architecture on a real dataset

✓Apply self-supervised pretraining to reduce labeling requirements

✓Implement a fusion mechanism to combine modalities into a single system

✓Evaluate quality using modality-specific metrics (mAP, WER, FID, and others)

Key topics

•Specifics of image/audio processing: spatial and temporal structures

•Backbone architectures: ResNet, ViT, EfficientNet (CV) / wav2vec, Whisper (Audio)

•Object detection (YOLO, DETR) or end-to-end ASR systems

•Segmentation (Mask R-CNN, SAM) or TTS (FastSpeech, VITS)

•Vision-language models: CLIP, BLIP, LLaVA / Audio-language: AudioLM, AudioPaLM

•Multimodal fusion: early, late, cross-attention

•Self-supervised learning within the chosen modality

•Augmentation and domain adaptation for CV/Audio

•Benchmarks and modality-specific evaluation metrics

This description was generated automatically and has not yet been reviewed by an instructor — it's a draft for discussion.

About the course

What you'll learn

Key topics

Feedback