lesson

- Fine-tune CLIP to classify car damage using real-world image categories - Use Google Custom Search API to generate labeled datasets from scratch - Apply PEFT techniques like LoRA to vision models and optimize hyperparameters with Optuna - Evaluate accuracy using cosine similarity over natural language prompts (e.g. “a car with large damage”) - Deploy the model in a real-world insurance agent workflow using LLaMA for reasoning over predictions

Dr. Dipen

Zao Yang

lesson

- Understand how CLIP learns joint image-text representations using contrastive learning - Run your first CLIP similarity queries and interpret shared embedding space - Practice prompt engineering with images — and see how wording shifts retrieval results - Build retrieval systems: text-to-image and image-to-image using cosine similarity - Experiment with visual vector arithmetic: apply analogies to embeddings - Explore advanced tasks like visual question answering (VQA) and image captioning - Compare multimodal architectures: CLIP, ViLT, ViT-GPT2 and how they process fusion - Learn how modality-specific encoders (image/audio) integrate into transformer models

Dr. Dipen

Zao Yang

lesson

Victor Engmark