Qualcomm Interview Prep — 06. Machine Learning & Deep Learning¶

Scope. Neural-network and deep-learning theory and practice as Qualcomm asks it — the neuron and forward pass, backpropagation (dense + convolutional, including writing it in C), gradient descent and optimizers, activations and losses, the convolution operation, 1×1 convolutions, pooling, batch normalization, the canonical CNN families (VGG / Inception / ResNet), overfitting and regularization, classification metrics, vanishing/exploding gradients, classic-vs-deep CV pipelines, signal-processing-for-ML, transformers/attention/GenAI, and quantization + on-device inference on the Hexagon NPU. This file owns the math and model side. The image-processing/ISP pipeline, 3A, HDR, denoise, demosaic live in 05_camera_isp_multimedia.md (cross-linked, not duplicated). Pure DSA (linked lists, trees, bit tricks) lives in 03_dsa.md; C memory/pointer mechanics behind the C-backprop answer live in 01_c_programming.md; OS/threading behind training-pipeline parallelism in 04_os.md; SoC/accelerator/number-format depth in 09_computer_arch_digital_design.md; behavioral "explain your ML project" framing in 11_behavioral_hr_projects.md.

How to read each entry. Every question is answered in layers so you can stop at the depth you need: - Q — the question, phrased as interviewers actually ask it. - Frequency — how often it showed up in the 75 collected reports (tier + approximate count). - Concept — the basis — book-style fundamentals with worked examples and, where it helps, an SVG diagram. - The "wh"s — Why it exists (what problem it solves), Where you see it (real Qualcomm camera/ISP/SoC/audio situations), and any important caveat. - Answer — a tight, say-it-out-loud interview answer. - Solution / good example — for "implement / design / derive X" questions, a complete, copy-pasteable code or derivation. - Follow-ups / gotchas — the traps interviewers spring next. - Seen in — the source reports.

Terms in bold-italics like backpropagation, ReLU, receptive field, softmax, quantization are defined in the § Encyclopedia at the bottom — search there for any keyword.

Frequency legend (sample = 75 collected interview reports; counts are approximate and partly from aggregator pages, so treat them as directional): 🔥🔥🔥 Very common (~8+ reports) · 🔥🔥 Common (~4–7) · 🔥 Occasional (~2–3) · ◽ Foundational (rarely asked verbatim, but assumed and underpinning everything else).

Why ML/DL shows up in the Qualcomm loop. Qualcomm ships the on-device AI brain of nearly every Android phone — the Hexagon NPU / Qualcomm AI Engine — plus camera computational-photography, computer-vision, audio-wake-word, and modem ML. The ML/CV/audio-ML loops (and the "ML & System Engineer" hybrid) probe whether you can derive backprop from the chain rule, reason about why ResNet trains, do a 1×1-conv FLOP count, and connect models to quantized, low-power, real-time inference on a mobile SoC. The signature observation from the evidence: candidates are asked to derive and even write backprop in C, and to explain VGG/Inception/ResNet/1×1-conv in the same breath as embedded systems questions. Evidence base: qualcomm_camera_interview_experiences.md.

Table of contents¶

• A. Neurons, forward pass & backprop — A1 perceptron/neuron & forward pass · A2 backprop in a dense net (chain-rule derivation) · A3 backprop in a CNN · A4 writing backprop in C
• B. Training: optimization, activations, losses — B1 gradient descent (batch/SGD/mini-batch) & LR · B2 optimizers (SGD-momentum/Adam) · B3 activation functions & dead ReLU · B4 loss functions (MSE vs cross-entropy) · B5 end-to-end training pipeline
• C. Convolution & CNN building blocks — C1 the convolution op, stride/padding/receptive field · C2 1×1 conv & FLOP reduction · C3 pooling · C4 batch normalization
• D. CNN architectures — D1 VGG · D2 Inception · D3 ResNet (and the family compared)
• E. Generalization & evaluation — E1 overfitting/regularization (L2/dropout) · E2 bias–variance · E3 precision/recall/F1/confusion matrix · E4 vanishing/exploding gradients
• F. Computer vision & signal processing for ML — F1 classic vs deep CV pipeline · F2 object detection (R-CNN/YOLO) · F3 signal-processing basics for ML (STFT/mel, CNN on audio)
• G. Generative AI, transformers & on-device inference — G1 transformer & attention · G2 LSTM vs Transformer · G3 quantization & on-device inference (Hexagon NPU)
• § Encyclopedia — searchable glossary
• § Last-5-minutes cheat sheet

A. Neurons, forward pass & backprop¶

A1 · Q: What is a perceptron / an artificial neuron, and what happens in the forward pass?¶

Frequency: ◽ Foundational — every ML/CV/audio round assumes it; it underpins A2–A4 and all of B/C/D.

Concept — the basis. A neuron computes a weighted sum of its inputs plus a bias, then squashes the result through a nonlinear activation function. With inputs x = [x₁…xₙ], weights w, and bias b:

z = w·x + b = Σ wᵢ xᵢ + b      (the "pre-activation" / logit)
a = σ(z)                        (the "activation" / output)

A single neuron with a step/sign activation is Rosenblatt's perceptron — a linear classifier (it can only separate linearly-separable data; famously it cannot learn XOR). Stack neurons into layers and the forward pass of a dense (fully-connected) network is just repeated matrix-multiply-then-activate:

a⁰ = x
zˡ = Wˡ aˡ⁻¹ + bˡ
aˡ = σ(zˡ)            for layers l = 1 … L;   ŷ = aᴸ

Worked example (one neuron, 2 inputs, sigmoid):

x = [1.0, 2.0], w = [0.5, -0.3], b = 0.1
z = 0.5·1.0 + (-0.3)·2.0 + 0.1 = 0.0
a = sigmoid(0) = 1/(1+e^0) = 0.5

Why it exists. It's the smallest unit that combines a learnable linear map (the weights — what the network adapts) with a fixed nonlinearity (the activation — what gives the network expressive power). Without the nonlinearity, stacking layers collapses to a single linear map (composition of linear maps is linear), so a deep net would be no more powerful than one layer. The nonlinearity is what makes "deep" worth anything.

Where you see it (Qualcomm). Every layer of a camera scene-classifier, a wake-word detector, or a super-resolution net is neurons/conv-filters doing exactly this. On the Hexagon NPU the w·x+b is a fused multiply-accumulate (MAC) — the tensor accelerator is essentially a giant MAC array, and the activation is a cheap pointwise op.

Answer. "A neuron takes a weighted sum of its inputs plus a bias, z = w·x + b, then applies a nonlinear activation a = σ(z). A perceptron is one such neuron with a threshold activation — a linear classifier. The forward pass of a network just repeats this layer by layer: multiply by the weight matrix, add the bias, apply the activation, feed forward. The nonlinearity is essential — without it, stacked layers collapse to a single linear transform."

Follow-ups / gotchas. A perceptron can't solve XOR (not linearly separable) — that needs a hidden layer (the historical "XOR problem"). The bias shifts the decision boundary off the origin. "MLP" = multi-layer perceptron = dense feed-forward net. Weights are the learned parameters; the activation is fixed.

Seen in: Report 11 (ML & System Engineer — backprop/CNN block assumes the neuron), Report 21 (Audio ML — "end-to-end training pipeline"), Report 5 (CV Engineer — CNNs). Standard ML-fundamentals expectation.

A2 · Q: Derive backpropagation for a dense network. (Walk me through the chain rule and the gradients.)¶

Frequency: 🔥🔥 Common (~4 reports) — explicitly asked in the ML & System Engineer loop ("Back-propagation algorithm in Dense Network") and assumed in every CV/audio-ML round.

Concept — the basis. Backpropagation is just the chain rule applied to the network's computation graph, run backwards from the loss, reusing intermediate results so the whole gradient costs about the same as one forward pass. Training = forward pass (compute ŷ and loss L) → backward pass (compute ∂L/∂W, ∂L/∂b for every layer) → gradient-descent update.

Define the error signal of layer l as δˡ = ∂L/∂zˡ (gradient of loss w.r.t. that layer's pre-activations). The four backprop equations:

(1) output layer:   δᴸ = ∂L/∂aᴸ ⊙ σ′(zᴸ)          ⊙ = elementwise product
(2) propagate back:  δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ σ′(zˡ)     (this is the recursion)
(3) weight grads:    ∂L/∂Wˡ = δˡ (aˡ⁻¹)ᵀ
(4) bias grads:      ∂L/∂bˡ = δˡ

Then update each parameter: Wˡ ← Wˡ − η·∂L/∂Wˡ.

Why the product? For a path L ← aᴸ ← zᴸ ← aᴸ⁻¹ ← zᴸ⁻¹ …, the chain rule says ∂L/∂zˡ = ∂L/∂zˡ⁺¹ · ∂zˡ⁺¹/∂aˡ · ∂aˡ/∂zˡ. The middle factor is Wˡ⁺¹ (since zˡ⁺¹ = Wˡ⁺¹aˡ + bˡ⁺¹), and the last is σ′(zˡ) — giving equation (2).

Worked micro-example (1 input → 1 hidden → 1 output, all scalars, sigmoid σ, MSE loss L = ½(ŷ−y)²):

forward:  z1 = w1·x + b1 ; a1 = σ(z1) ; z2 = w2·a1 + b2 ; ŷ = σ(z2)
backward: δ2 = (ŷ − y)·σ′(z2)            σ′(z) = σ(z)(1−σ(z))
          ∂L/∂w2 = δ2·a1     ∂L/∂b2 = δ2
          δ1 = (w2·δ2)·σ′(z1)
          ∂L/∂w1 = δ1·x      ∂L/∂b1 = δ1

Why it exists. A deep net has millions of parameters; computing each ∂L/∂wᵢ independently (finite differences) would need a forward pass per parameter — hopelessly slow. Backprop computes all gradients in one backward sweep by caching the forward activations and reusing the downstream δ. It's reverse-mode automatic differentiation, and it's what makes training deep nets tractable at all.

Where you see it (Qualcomm). Training happens off-device (datacenter), but understanding backprop is what lets you reason about why a model won't converge, why gradients vanish in a deep camera net, and what a framework like PyTorch/QNN is doing under loss.backward(). It's the theory behind quantization-aware training (QAT) used to prep models for the NPU.

Answer. "Backprop is the chain rule on the network's computation graph, computed backwards from the loss so we reuse intermediate results. I define δˡ = ∂L/∂zˡ. At the output, δᴸ = ∂L/∂aᴸ ⊙ σ′(zᴸ). Then I recurse backward: δˡ = (Wˡ⁺¹)ᵀ δˡ⁺¹ ⊙ σ′(zˡ) — the transposed weight matrix moves the error to the previous layer, and σ′ accounts for the activation. The parameter gradients are ∂L/∂Wˡ = δˡ(aˡ⁻¹)ᵀ and ∂L/∂bˡ = δˡ. Finally gradient descent: W ← W − η ∂L/∂W. The whole backward pass costs about one forward pass."

Follow-ups / gotchas. With softmax + cross-entropy at the output, the messy σ′ cancels and δᴸ = ŷ − y exactly (a key simplification — see B4). The ⊙ σ′(zˡ) factor is exactly where vanishing gradients come from (E4): if σ′ < 1 repeatedly, δ shrinks geometrically through depth — the reason ReLU (σ′=1 for z>0) and ResNet skip connections exist. You must cache the forward zˡ/aˡ to do the backward pass.

Seen in: Report 11 ("Back-propagation algorithm in Dense Network"). Standard deep-learning expectation across Reports 5, 21.

A3 · Q: How does backpropagation work in a convolutional network? (How is it different from a dense layer?)¶

Frequency: 🔥🔥 Common (~3 reports) — explicitly asked in the ML & System Engineer loop ("Back-propagation in Convolution Network").

Concept — the basis. A conv layer is still a linear-op-then-activation, so the same four backprop equations apply — but the linear op is convolution with shared weights, which changes the shape of the gradients:

1. Forward: Z = X ∗ K + b (cross-correlation in practice), then A = σ(Z). The kernel K is shared across all spatial positions (weight tying).
2. Gradient w.r.t. the kernel (∂L/∂K): because each kernel weight touches every output position, its gradient is the sum over all positions — which works out to a convolution of the input X with the incoming gradient δ (∂L/∂K = X ∗ δ).
3. Gradient w.r.t. the input (∂L/∂X, to pass back to the previous layer): a "full" convolution of δ with the 180°-rotated (flipped) kernel (∂L/∂X = δ ∗ rot180(K)). Intuitively, error flows back through the same connections that carried the forward signal.
4. Pooling layers have no weights; backprop just routes the gradient: max-pool sends the whole gradient to the position that was the max (others get 0); average-pool spreads it equally.

Shape sanity check (no padding, stride 1): forward out = (W − K)/S + 1 (C1). The ∂L/∂X "full convolution" restores the larger input size; the ∂L/∂K convolution yields a kernel-sized gradient.

Why it exists / why it differs. Two CNN tricks change the bookkeeping versus a dense layer: weight sharing (so one kernel's gradient accumulates contributions from every spatial location) and local connectivity (so the input gradient is a sparse, structured convolution rather than a dense matrix-vector product). The payoff is far fewer parameters and translation-equivariance — but you must remember to sum the kernel gradient over space and flip the kernel for the input gradient.

Where you see it (Qualcomm). Conceptually behind training every camera/CV CNN (denoise, segmentation, detection). The "flip + full-conv" structure is also why the forward pass can be expressed as matrix multiply (im2col) — the trick the Hexagon NPU/HVX uses to map convolution onto a MAC array.

Answer. "It's the same chain-rule backprop, but the linear operation is a shared-weight convolution instead of a dense matmul. Three differences: (1) the kernel gradient is a convolution of the layer input with the upstream gradient, summed over all spatial positions because the weights are tied; (2) the gradient w.r.t. the input is a full convolution of the upstream gradient with the 180°-flipped kernel, routing error back through the same local connections; (3) pooling layers have no parameters — max-pool routes the gradient only to the argmax position, average-pool spreads it evenly. ReLU just gates the gradient where the forward input was positive."

Solution / good example — gradient routing through max-pool (the part people forget):

# forward: remember WHICH index won, per 2x2 window
out[i,j] = max(window);  argmax_idx[i,j] = position_of_max
# backward: gradient goes only to that winner
dX[:] = 0
dX[argmax_idx[i,j]] += dOut[i,j]      # all other cells in the window get 0

Follow-ups / gotchas. "Convolution" in DL is really cross-correlation (no kernel flip in the forward pass); the flip reappears in the backward input-gradient. Stride > 1 means the input-gradient convolution uses a dilated/upsampled δ. Don't forget the bias gradient is the sum of δ over spatial positions (and batch). This connects to A4 (writing it in C).

Seen in: Report 11 ("Back-propagation in Convolution Network"). Standard CNN expectation.

A4 · Q: Write a C program for backpropagation in a convolutional network. (At least sketch the structure.)¶

Frequency: 🔥 Occasional (~1–2 reports) — but a distinctive, literally-asked Qualcomm question (the ML & System Engineer loop says "Write a C program for Back-propagation in Conv Network"). They blend ML with embedded-C on purpose.

Concept — the basis. The interviewer is testing whether you can turn the math of A3 into concrete loops with correct indexing and memory — exactly the C skills from 01_c_programming.md applied to tensors. Tensors are stored as flat arrays with manual index arithmetic; there's no autodiff, so you write the forward and backward loops by hand. Structure:

• Data layout: a feature map [C][H][W] flattened to data[c*H*W + h*W + w] (row-major); a kernel [Cout][Cin][KH][KW].
• Forward conv: 6 nested loops (out-channel, out-y, out-x, in-channel, ky, kx) accumulating MACs.
• Backward: (a) dK += input-patch · upstream-grad (summed over positions); (b) dX += flipped-kernel · upstream-grad (full conv); (c) apply the activation derivative (gate by ReLU mask).

Why it's asked. It fuses two things Qualcomm cares about: do you understand backprop concretely (not just "call .backward()"), and can you write careful, correct, memory-safe C with nested-loop index math — the everyday job in NPU kernels, DSP code, and HAL. It's the ML version of "implement memcpy."

Where you see it (Qualcomm). Hand-written/optimized convolution and gradient kernels for the Hexagon DSP/HVX; reference C implementations used to validate a hardware accelerator's output (bit-exact model-vs-hardware comparison — exactly the "compare output between hardware and software models" theme in Report 3).

Answer. "I'd store tensors as flat float arrays with explicit index math, write the forward conv as nested loops accumulating multiply-adds, cache the pre-activations, then write three backward pieces: the ReLU mask gating the upstream gradient, the kernel gradient as input-patch times upstream-gradient summed over all output positions, and the input gradient as a full convolution of the upstream gradient with the flipped kernel. Finally an SGD update K -= lr*dK. I'd keep it single-channel and stride-1 first for clarity, then generalize."

Solution / good example — a minimal but complete single-channel conv layer with backprop in C:

#include <string.h>
/* Single input channel, single 3x3 kernel, stride 1, valid (no) padding.
   IN: HxW input X; 3x3 kernel K; OUT: (H-2)x(W-2) output.
   Forward:  Z = X (cross-)correlate K ;  A = relu(Z)
   Backward: given dA (grad wrt A), produce dK and dX, and the relu mask. */
#define H 5
#define W 5
#define KS 3
#define OH (H-KS+1)
#define OW (W-KS+1)

static float relu(float v){ return v > 0 ? v : 0.f; }

/* ---- forward ---- */
void conv_forward(const float X[H][W], const float K[KS][KS], float b,
                  float Z[OH][OW], float A[OH][OW]) {
    for (int oy = 0; oy < OH; oy++)
        for (int ox = 0; ox < OW; ox++) {
            float acc = b;
            for (int ky = 0; ky < KS; ky++)
                for (int kx = 0; kx < KS; kx++)
                    acc += X[oy+ky][ox+kx] * K[ky][kx];   /* MAC */
            Z[oy][ox] = acc;
            A[oy][ox] = relu(acc);
        }
}

/* ---- backward ----
   dA  : upstream gradient wrt A   (OH x OW)
   dZ  : = dA * relu'(Z)           (relu' is 1 where Z>0 else 0)
   dK  : grad wrt kernel  (sum over all output positions)
   dX  : grad wrt input   (full conv of dZ with flipped K) */
void conv_backward(const float X[H][W], const float K[KS][KS], const float Z[OH][OW],
                   const float dA[OH][OW], float dK[KS][KS], float *db, float dX[H][W]) {
    float dZ[OH][OW];
    memset(dK, 0, sizeof(float)*KS*KS);
    memset(dX, 0, sizeof(float)*H*W);
    *db = 0.f;

    /* 1) gate by ReLU derivative */
    for (int oy = 0; oy < OH; oy++)
        for (int ox = 0; ox < OW; ox++)
            dZ[oy][ox] = (Z[oy][ox] > 0.f) ? dA[oy][ox] : 0.f;

    /* 2) accumulate dK, db, and dX in one sweep */
    for (int oy = 0; oy < OH; oy++)
        for (int ox = 0; ox < OW; ox++) {
            float g = dZ[oy][ox];
            *db += g;                                   /* bias grad = sum of dZ */
            for (int ky = 0; ky < KS; ky++)
                for (int kx = 0; kx < KS; kx++) {
                    dK[ky][kx] += X[oy+ky][ox+kx] * g;  /* dK = X (corr) dZ      */
                    dX[oy+ky][ox+kx] += K[ky][kx] * g;  /* dX = full-conv route  */
                }
        }
}

/* ---- SGD update ---- */
void sgd_update(float K[KS][KS], float *b, const float dK[KS][KS], float db, float lr) {
    for (int i = 0; i < KS; i++)
        for (int j = 0; j < KS; j++)
            K[i][j] -= lr * dK[i][j];
    *b -= lr * db;
}

Follow-ups / gotchas. Use float (or int for a quantized version), watch for out-of-bounds indexing (the classic C bug — 01_c_programming.md), and memset the gradient buffers to zero before accumulating (they're summed, not assigned). The scatter form of dX avoids a separate flipped-kernel buffer. Mention im2col + matmul as the production-fast path.

Seen in: Report 11 ("Write a C program for Back-propagation in Conv Network"). A distinctive Qualcomm ML-meets-C question.

B. Training: optimization, activations, losses¶

B1 · Q: Explain gradient descent — batch vs SGD vs mini-batch — and how the learning rate matters.¶

Frequency: 🔥🔥 Common (~3–4 reports) — assumed in every ML/CV/audio round; the training-pipeline question (Report 21) drills it.

Concept — the basis. Gradient descent minimizes the loss L(θ) by repeatedly stepping opposite the gradient (the direction of steepest increase), scaled by the learning rate η:

θ ← θ − η · ∇L(θ)

The three flavors differ in how much data each step uses to estimate ∇L:

Worked example (1 step, 1 param, MSE): L = ½(w·x − y)², so ∂L/∂w = (w·x − y)·x. With x=2, y=6, w=1, η=0.1: pred =2, error =−4, grad =−8, update w ← 1 − 0.1·(−8) = 1.8 (moving toward the true w=3).

Learning-rate effects: η too small → painfully slow, may stall; η too large → overshoots, oscillates, or diverges (loss → ∞/NaN); a good η descends fast then settles. In practice you use an LR schedule (warmup + decay) and adaptive optimizers (B2).

Why it exists. The loss surface of a neural net has no closed-form minimum, so you find it iteratively. Computing the exact gradient over millions of examples every step (batch GD) is too slow and memory-hungry; using one example (SGD) is cheap but noisy. Mini-batch is the sweet spot — enough samples for a stable gradient, small enough to vectorize on parallel hardware, and the noise even helps generalization.

Where you see it (Qualcomm). Training the camera/CV/audio models that later get quantized for the NPU; batch size is tuned to the training accelerator's memory; the inference side (what runs on Hexagon) is just the forward pass — no gradient descent on-device (except niche on-device personalization).

Answer. "Gradient descent steps parameters opposite the loss gradient, θ ← θ − η∇L. Batch GD uses the whole dataset per step — exact but slow; stochastic GD uses one sample — fast and noisy; mini-batch uses a small batch — the practical default because it gives a stable gradient and maps efficiently onto GPU/NPU vectorized hardware. The learning rate sets step size: too small is slow and may stall, too large overshoots and can diverge to NaN. I'd use mini-batch with an LR schedule, typically warmup then decay."

Follow-ups / gotchas. Mini-batch noise is a feature (regularization, escaping saddle points). Larger batches → less noise but often need a larger LR (the "linear scaling rule"). Diverging loss / NaN almost always means the LR is too high (or no input normalization). One "epoch" = one full pass over the data = many mini-batch steps.

Seen in: Report 21 ("optimizers" in the training pipeline), Report 5 ("optimization"). Standard ML expectation.

B2 · Q: Explain an optimizer like Adam (vs plain SGD / momentum).¶

Frequency: 🔥 Occasional (~1–2 reports) — explicitly in the Audio-ML loop ("Explain the Adam optimizer… first/second moments, bias correction, lr=1e-3, betas=(0.9,0.999)").

Concept — the basis. Plain SGD uses the raw gradient; smarter optimizers adapt the step.

• SGD + momentum: accumulate a velocity v ← βv + (1−β)g and step with v. Like a heavy ball rolling downhill — damps oscillation across ravines, accelerates along consistent directions.
• Adam (Adaptive Moment Estimation): keeps two running averages per parameter — the first moment m (mean of gradients, like momentum) and the second moment v (mean of squared gradients, a per-parameter variance). It then does bias correction (because m,v start at 0 and are biased early) and takes a step that's normalized by the per-parameter gradient magnitude:

m ← β₁ m + (1−β₁) g           v ← β₂ v + (1−β₂) g²
m̂ = m/(1−β₁ᵗ)                 v̂ = v/(1−β₂ᵗ)        (bias correction at step t)
θ ← θ − η · m̂ / (√v̂ + ε)

Why it exists. A single global learning rate is wrong for parameters with very different gradient scales. Adam gives each parameter its own effective step size (small steps where gradients are large/noisy, larger where they're small/consistent), and momentum smooths the trajectory — so it converges fast and robustly with little tuning. That's why it's the default for most deep nets.

Where you see it (Qualcomm). Training the audio wake-word / camera models that get deployed to the NPU; the Adam state (m, v) is training-time only and discarded before quantization/export — only the final weights ship to the device.

Answer. "Adam tracks two per-parameter running averages: the first moment m (mean gradient, i.e. momentum) and the second moment v (mean squared gradient). It bias-corrects them — m̂ = m/(1−β₁ᵗ), v̂ = v/(1−β₂ᵗ) — since they start at zero, then steps θ −= η·m̂/(√v̂+ε). So each parameter gets an adaptive step normalized by its gradient magnitude, with momentum smoothing. Defaults: lr 1e-3, betas (0.9, 0.999), eps 1e-8. It's fast and robust with little tuning; well-tuned SGD-with-momentum can generalize slightly better, which is why some vision models still use it."

Follow-ups / gotchas. Bias correction matters most in the first few steps. β₂=0.999 means v averages over ~1000 steps. AdamW decouples weight decay from the gradient (better L2 regularization). Adam's per-parameter state doubles+ the optimizer memory — irrelevant at inference, relevant for training-memory budgeting.

Seen in: Report 21 ("Explain the Adam optimizer (first/second moments, bias correction…)"). Standard training expectation.

B3 · Q: What activation functions do you know? Why ReLU? What is the "dying ReLU" problem?¶

Frequency: 🔥🔥 Common (~3 reports) — implicit in every CNN/training discussion (Reports 5, 11, 21).

Concept — the basis. The activation function is the per-neuron nonlinearity. The main ones:

Why ReLU. Sigmoid/tanh saturate: for large |z| their derivative → 0, so in a deep net the ⊙ σ′(z) factor in backprop (A2) drives gradients to zero — vanishing gradients (E4), and training stalls in early layers. ReLU has derivative 1 for all positive inputs, so gradients pass through undiminished; it's also just a max (no exponential), so it's far cheaper. These two properties — non-saturating gradient + cheap compute — are why ReLU made very deep CNNs trainable.

Dying ReLU. Because ReLU outputs 0 (and has gradient 0) for any negative input, a neuron that gets pushed into the negative region — e.g. by a large gradient step or a big negative bias — can output 0 for every input forever, and since its gradient is also 0 it never recovers. That neuron is "dead." Fixes: Leaky ReLU / PReLU / ELU (nonzero negative slope keeps a gradient alive), careful initialization, and lower learning rates.

Why it exists. The nonlinearity is what gives the network expressive power (A1). The choice trades smoothness, gradient health, compute cost, and output range. The field moved sigmoid → tanh → ReLU → ReLU-variants precisely to fix gradient flow and speed.

Where you see it (Qualcomm). ReLU/ReLU6 dominate mobile CNNs because they're trivially cheap and quantization-friendly — ReLU6 (min(max(0,z),6)) caps activations to a known range, which makes fixed-point/INT8 quantization on the Hexagon NPU clean (a bounded range maps neatly to 8 bits). Softmax sits at the output of a classifier.

Answer. "Sigmoid and tanh are smooth but saturate — their derivative vanishes for large inputs, which causes vanishing gradients in deep nets, so training stalls. ReLU, max(0,z), has gradient 1 on the positive side so gradients flow undiminished, and it's just a max so it's cheap — that's why it's the CNN default. Its downside is the dying-ReLU problem: a neuron knocked into the negative region outputs 0 with 0 gradient and can never recover. Leaky ReLU adds a small negative slope to keep a gradient alive and fix that. On mobile we like ReLU6 because the bounded range quantizes cleanly to INT8."

Follow-ups / gotchas. Softmax is for the output (probabilities), not hidden layers. Sigmoid is still used for binary classification and gates (LSTM). Tanh is zero-centered (often better than sigmoid for hidden units, historically). Dying ReLU is worsened by high LR. GELU/Swish are common in transformers.

Seen in: Reports 5, 11, 21 (CNN training / architectures imply activation choices). Standard deep-learning expectation.

B4 · Q: When do you use MSE vs cross-entropy loss? Why is cross-entropy better for classification?¶

Frequency: 🔥 Occasional (~1–2 reports) — part of the training-pipeline / "loss functions" probe (Report 21) and CV model-selection (Report 5).

Concept — the basis. The loss function is the scalar the network minimizes.

• MSE (Mean Squared Error): L = (1/N) Σ (ŷ − y)². The right loss for regression (predicting a continuous value — pixel intensity, depth, a bounding-box coordinate).
• Cross-entropy (CE): for classification with softmax outputs, L = − Σ yᵢ log(ŷᵢ) (binary version −[y log ŷ + (1−y) log(1−ŷ)]). The right loss for classification (predicting a probability distribution over classes).

Why CE is better for classification. Two reasons: 1. It's the right objective. CE is the negative log-likelihood of the correct class under the predicted distribution — minimizing it = maximizing the probability the model assigns to the truth. MSE on probabilities doesn't correspond to a sensible likelihood and under-penalizes confident wrong answers. 2. Healthy gradients (the big one). With sigmoid/softmax + MSE, the gradient contains a σ′(z) factor that vanishes when the output saturates — so a confidently wrong neuron (output ≈ 1 when it should be 0) learns slowest, exactly backwards. With sigmoid/softmax + cross-entropy, the log cancels the exp in the activation and the output-layer gradient collapses to the clean δ = ŷ − y. The gradient is proportional to the error, so confident mistakes produce large corrective gradients. This is the key practical reason CE trains classifiers faster.

Derivation sketch (sigmoid + binary CE):

ŷ = σ(z),  L = −[y log ŷ + (1−y) log(1−ŷ)]
∂L/∂z = ŷ − y        ← the σ′(z) cancels cleanly; no saturation stall

Why it exists. Different tasks need different "what does wrong mean" measures. Squared error is the natural distance for real values; cross-entropy is the natural divergence between probability distributions. Matching loss to task (and to the output activation) is what makes training well-behaved.

Where you see it (Qualcomm). Cross-entropy for a camera scene/object classifier or an audio wake-word detector; MSE (or L1/Charbonnier) for regression-style vision — super-resolution, denoising, depth, bounding-box regression. Detection losses combine both (CE/focal for class + smooth-L1/IoU for box).

Answer. "MSE for regression — continuous targets like pixel values, depth, or box coordinates. Cross-entropy for classification — it's the negative log-likelihood of the correct class, so it directly maximizes the probability of the truth. The crucial reason CE wins for classification: with softmax/sigmoid plus MSE, the gradient has a σ′(z) factor that vanishes when the output saturates, so a confidently-wrong neuron learns slowest. With cross-entropy the log cancels the exp and the output gradient becomes simply ŷ − y — proportional to the error — so confident mistakes get strong gradients and training is faster and more stable."

Follow-ups / gotchas. Always pair softmax with cross-entropy (and use the numerically-stable fused log-softmax + NLL, not softmax-then-log, to avoid overflow). For class imbalance use weighted CE or focal loss. Label smoothing regularizes CE. For detection/segmentation, CE/Dice/IoU variants are standard.

Seen in: Report 21 ("loss functions" in the pipeline), Report 5 ("model selection, evaluation"). Standard expectation.

B5 · Q: Walk me through the end-to-end training pipeline for a neural network.¶

Frequency: 🔥 Occasional (~1–2 reports) — literally the opening question of the Audio-ML loop (Report 21), and the spine of the CV "dataset-to-deployment" walkthrough (Reports 5, 10).

Concept — the basis. A complete supervised training pipeline, in order:

1. Data           collect, clean, label; split train/val/test (e.g. 80/10/10)
2. Preprocess     normalize/standardize inputs; for audio: STFT → mel-spectrogram (F3)
3. Augment        cheap label-preserving perturbations to fight overfitting (E1)
                  images: flip, crop, color-jitter;  audio: noise/reverb/pitch/time-stretch,
                  SpecAugment, SNR mixing (e.g. MUSAN)
4. Batch          shuffle; form mini-batches (B1); feed via a DataLoader
5. Model          define architecture (CNN/ResNet/transformer …)
6. Loss           pick MSE vs cross-entropy to match the task (B4)
7. Optimizer      Adam/SGD-momentum + LR schedule (B1, B2)
8. Train loop     for each batch: forward → loss → backward (A2) → optimizer.step()
9. Validate       track val metric each epoch; early-stop on plateau; checkpoint best
10. Evaluate      final metrics on the held-out TEST set (precision/recall/F1, E3)
11. Deploy        export → quantize → run on target (NPU) (G3)

Why it exists. It's the disciplined loop that turns raw data into a deployable model while not fooling yourself: the train/val/test split and held-out test set guard against overfitting (E1); augmentation cheaply multiplies data; the validation metric drives early stopping and model selection; only at the very end do you touch the test set.

Where you see it (Qualcomm). Exactly the audio-ML scenario in Report 21: "use a CNN on spectrogram inputs for Alexa wake-word recordings; augment clean audio with noise/reverb/pitch-shift/time-stretch (MUSAN, SNR mixing, SpecAugment)." The pipeline ends at the Qualcomm-specific step the others don't have: quantize and deploy to the Hexagon NPU (G3).

Answer. "Data → preprocess → augment → batch → model → loss → optimizer → train loop → validate → test → deploy. Concretely: split into train/val/test, normalize inputs (for audio, convert to a mel-spectrogram), apply label-preserving augmentation to fight overfitting, shuffle into mini-batches, then loop forward-pass → compute loss → backprop → optimizer step. Each epoch I validate, early-stop on a plateau, and checkpoint the best model. I only evaluate on the held-out test set at the end with the real metric — precision/recall/F1 for a detector. Finally I export and quantize the model to run efficiently on the device."

Solution / good example — a minimal, correct PyTorch training loop (the "memorize this" template):

model.train()
for epoch in range(num_epochs):
    for x, y in train_loader:               # mini-batches (shuffled)
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()               # clear stale grads (they accumulate!)
        logits = model(x)                   # forward
        loss = criterion(logits, y)         # e.g. nn.CrossEntropyLoss()
        loss.backward()                     # backprop (A2)
        optimizer.step()                    # update params (B1/B2)
    validate(model, val_loader)             # early-stop / checkpoint on val metric

Follow-ups / gotchas. Forgetting optimizer.zero_grad() makes gradients accumulate across batches (a classic bug). Never tune on the test set (data leakage). Normalize using train-set statistics only. Augment only the training data, not val/test. Track a task metric (F1/mAP), not just loss.

Seen in: Report 21 ("end-to-end training pipeline… data prep, augmentation, feature extraction, batching, loss functions, optimizers, evaluation"), Reports 5 & 10 ("dataset preparation to deployment"). Standard ML expectation.

C. Convolution & CNN building blocks¶

C1 · Q: Explain the convolution operation. What are stride, padding, and the receptive field?¶

Frequency: 🔥🔥 Common (~3 reports) — the bedrock of every CNN question (Reports 5, 11) and the CV pipeline (Report 3).

Concept — the basis. A convolution layer slides a small learnable kernel (filter) over the input feature map; at each position it computes the dot product of the kernel with the overlapping patch (plus bias), producing one output pixel. Sliding the kernel across all positions builds the output feature map. Each filter learns to detect one local pattern (edge, texture, blob); a layer has many filters → many output channels.

• Stride (S): how many pixels the kernel jumps each step. S=1 dense; S=2 downsamples by ~2× (cheaper, smaller output).
• Padding (P): zeros added around the border. "Valid" = no padding (output shrinks). "Same" = pad so the output keeps the input's spatial size.
• Output size: out = (W − K + 2P)/S + 1 per spatial dimension.
• Receptive field: the region of the original input that influences one output activation. It grows with depth — stacking small kernels (or adding stride/pooling) makes deep neurons "see" large parts of the image, which is how CNNs build from edges → textures → objects.

Worked example: 5×5 input, 3×3 kernel, stride 1, no padding → out = (5 − 3 + 0)/1 + 1 = 3 → a 3×3 output (the diagram). With "same" padding (P=1): out = (5 − 3 + 2)/1 + 1 = 5 (size preserved). Two stacked 3×3 convs give each output a 5×5 receptive field (the VGG insight, D1).

Why it exists. Three wins over a dense layer on images: (1) parameter sharing — one small kernel is reused everywhere, so a conv layer has far fewer weights than a fully-connected one (a 3×3 filter = 9 weights regardless of image size); (2) locality — vision features are local, so a neuron only needs to look at a small patch; (3) translation equivariance — the same feature is detected wherever it appears. These priors make CNNs sample-efficient and the natural fit for images.

Where you see it (Qualcomm). Every camera/CV CNN; the conv is the dominant compute on the Hexagon NPU, mapped to its MAC/tensor array (often via im2col → matrix-multiply). Stride and pooling control the resolution/compute trade-off in a real-time mobile vision pipeline.

Answer. "A conv layer slides a learnable kernel over the input, taking the dot product with each local patch plus a bias to produce one output pixel; many filters give many output channels, and each filter learns one local pattern like an edge. Stride is the step size (stride 2 downsamples), padding adds border zeros — 'same' padding preserves spatial size, 'valid' shrinks it. Output size is (W − K + 2P)/S + 1. The receptive field is how much of the original input one output sees; it grows with depth, which is how a CNN goes from edges to textures to whole objects. Convolution wins through parameter sharing, locality, and translation equivariance."

Follow-ups / gotchas. DL "convolution" is technically cross-correlation (no kernel flip). A K×K conv over C_in→C_out channels has K·K·C_in·C_out weights. Dilated convolution enlarges the receptive field without more parameters. Stride-2 conv vs pooling are two ways to downsample.

Seen in: Reports 5, 11 (CNNs/architectures), Report 3 (image-processing workflow, edge detection). Standard CNN expectation.

C2 · Q: How does a 1×1 convolution reduce computation? (Work the FLOPs.)¶

Frequency: 🔥🔥 Common (~2 reports) — literally asked in the ML & System Engineer loop ("How 1x1 Convolution reduces computation?"). A signature Qualcomm/Inception question.

Concept — the basis. A 1×1 convolution has a 1×1 spatial kernel, so it doesn't mix neighbors — it mixes channels. At each pixel it's a little fully-connected layer across the C_in channels, producing C_out channels. It keeps H×W unchanged but can shrink (or grow) the channel depth — a bottleneck. By cutting channel depth before an expensive 3×3 or 5×5 conv, it slashes the FLOPs of that conv.

Worked FLOP example (the canonical Inception number — verified):

Input: 28×28×192 feature map.   Goal: produce 28×28×32 via a 5×5 'same' conv.

(A) DIRECT 5×5, 192 → 32:
    cost = (output pixels) × (MACs per output)
         = (28·28·32) × (5·5·192)
         ≈ 120 million MACs (~120M FLOPs)

(B) BOTTLENECK: 1×1 (192→16), then 5×5 (16→32):
    1×1 stage: (28·28·16) × (1·1·192) ≈ 2.4M
    5×5 stage: (28·28·32) × (5·5·16)  ≈ 10.0M
    total      ≈ 12.4 million  → about 10× cheaper than (A)

Why it exists. Two jobs: (1) cheap dimensionality reduction — squeeze channels so the heavy spatial convs operate on a thin volume (the GoogLeNet/Inception bottleneck, also ResNet's bottleneck block); (2) cross-channel feature mixing + nonlinearity — a 1×1 conv + ReLU recombines channels and adds depth/expressivity cheaply ("network in network"). It buys most of the representational benefit of more channels at a fraction of the compute.

Where you see it (Qualcomm). Exactly the kind of efficiency trick that matters for mobile, power-constrained inference on the Hexagon NPU — fewer MACs = less energy and lower latency per frame. MobileNet's depthwise-separable convolution (depthwise + 1×1 pointwise) is the mobile-CNN workhorse and leans entirely on the 1×1 to combine channels cheaply.

Answer. "A 1×1 conv doesn't look at neighboring pixels — it mixes channels at each pixel, like a per-pixel fully-connected layer across channels, so it can squeeze the channel depth. Put a 1×1 bottleneck before a costly 5×5 and you compute the 5×5 on far fewer channels. Concretely, a 5×5 conv turning 28×28×192 into 28×28×32 costs about 120M MACs; insert a 1×1 that drops 192→16 first (≈2.4M) then 5×5 on 16→32 (≈10M) and the total is ~12.4M — roughly 10× cheaper, for essentially the same output. That's the Inception/ResNet bottleneck idea, and it's why MobileNet uses 1×1 pointwise convs for cheap mobile inference."

Follow-ups / gotchas. It only reduces compute when C_out < C_in (it's a bottleneck); a 1×1 can also expand channels. It adds a ReLU → extra nonlinearity. Param count of a 1×1 over C_in→C_out is just C_in·C_out (no spatial term). Depthwise-separable conv = depthwise (spatial, per-channel) + pointwise (1×1, cross-channel) — the mobile staple.

Seen in: Report 11 ("How 1x1 Convolution reduces computation?"). A distinctive, high-value Qualcomm question.

C3 · Q: What is pooling? Max vs average pooling.¶

Frequency: 🔥 Occasional (~1–2 reports) — assumed in the CNN-architecture discussions (Reports 5, 11).

Concept — the basis. Pooling downsamples a feature map by summarizing each small window (typically 2×2, stride 2) into one value — no learnable weights. - Max pooling: takes the maximum in each window → keeps the strongest activation (most salient feature), adds small translation invariance. The default in classic CNNs. - Average pooling: takes the mean → a smoother summary. Global average pooling (average the entire feature map to one value per channel) often replaces the final fully-connected layers (fewer params, less overfitting; used in Inception/ResNet).

Example: a 2×2 max-pool on [[1,3],[2,4]] → 4; average-pool → 2.5. A 2×2 stride-2 pool halves H and W (quarters the spatial size).

Why it exists. Three benefits: (1) reduce spatial resolution → fewer activations → less compute and memory downstream; (2) enlarge the receptive field quickly (C1); (3) small translation invariance — a feature shifting by a pixel still triggers the same pooled output. Max-pool keeps the presence of a feature regardless of exact position.

Where you see it (Qualcomm). Standard in mobile vision backbones to cut resolution cheaply; global average pooling shrinks the head, helping both accuracy and the model size that must fit the NPU. (Some efficient nets replace pooling with stride-2 convolutions instead.)

Answer. "Pooling downsamples a feature map by summarizing each window into one value, with no learned weights. Max pooling keeps the maximum — the strongest feature response — giving a bit of translation invariance and is the classic default. Average pooling takes the mean, a smoother summary; global average pooling reduces a whole channel to one number and often replaces fully-connected layers to cut parameters and overfitting. Pooling reduces compute/memory, grows the receptive field, and adds small shift-invariance."

Follow-ups / gotchas. Backprop through max-pool routes the gradient only to the argmax position (A3); average-pool spreads it evenly. Pooling is gradually being replaced by strided convs in some architectures. Pooling discards location info — bad for tasks needing precise localization (segmentation uses unpooling/transposed conv to recover it).

Seen in: Reports 5, 11 (CNN architecture). Standard CNN expectation.

C4 · Q: What is batch normalization and why does it help?¶

Frequency: 🔥 Occasional (~1–2 reports) — assumed in modern-CNN/training discussion (Reports 5, 11, 21).

Concept — the basis. Batch normalization normalizes a layer's pre-activations per mini-batch to zero mean and unit variance, then applies a learnable scale γ and shift β:

μ = mean(z over batch)        σ² = var(z over batch)
ẑ = (z − μ)/√(σ² + ε)         (normalize)
y = γ·ẑ + β                   (learnable scale + shift — lets the net undo normalization if useful)

Why it exists / why it helps. The original paper framed it as reducing internal covariate shift (each layer's input distribution keeps shifting as earlier layers update). The more accepted modern explanation (Santurkar et al., 2018) is that BN smooths the optimization landscape — it improves the Lipschitzness of the loss and gradients, so the loss changes more predictably and gradient descent can take larger, more stable steps. Practically, BN: lets you use higher learning rates, makes training less sensitive to initialization, speeds convergence, and adds a mild regularizing effect (the batch-statistic noise).

Where you see it (Qualcomm). Standard in training camera/CV CNNs (the ResNet/Inception blocks are conv→BN→ReLU). Crucially for Qualcomm: at deployment, BN is folded into the preceding conv (since both are linear at inference, BN(conv(x)) collapses into a single conv with adjusted weights/bias) — so it costs zero extra ops on the Hexagon NPU and quantizes cleanly.

Answer. "Batch norm normalizes each layer's pre-activations across the mini-batch to zero mean and unit variance, then applies a learnable scale and shift so the network can recover any distribution it actually needs. It was introduced to reduce internal covariate shift, but the better-supported explanation is that it smooths the loss landscape — making gradients more reliable so you can train with higher learning rates, faster convergence, and less sensitivity to initialization, plus a little regularization. At inference it uses running statistics, and in deployment we fold BN into the preceding convolution so it's free at runtime."

Follow-ups / gotchas. BN behaves differently in train vs eval mode (batch stats vs running stats) — forgetting model.eval() is a classic bug. It's weak for very small batches (noisy stats) → use LayerNorm (transformers/RNNs) or GroupNorm instead. BN sits before the activation (conv → BN → ReLU). Fold BN into conv for inference.

Seen in: Reports 5, 11, 21 (modern CNN/training). Standard deep-learning expectation.

D. CNN architectures¶

D1 · Q: Describe the VGG architecture. Why stacks of 3×3 convolutions?¶

Frequency: 🔥🔥 Common (~2 reports) — explicitly asked in the ML & System Engineer loop ("architectures of Inception, VGG").

Concept — the basis. VGG (Oxford VGGNet, 2014) is the "simplicity and uniformity" CNN: a deep stack (VGG-16/VGG-19) of only 3×3 convolutions (stride 1, same padding) and 2×2 max-pools, doubling channels after each pool (64→128→256→512), ending in a few fully-connected layers + softmax. No fancy modules — just many small convs.

The 3×3-stack insight. Two stacked 3×3 convs have the same receptive field as one 5×5 (C1), and three 3×3s match a 7×7 — but with fewer parameters and more nonlinearity:

one 7×7 conv (C→C):   7·7·C·C = 49C²  weights, 1 ReLU
three 3×3 convs:      3·(3·3·C·C) = 27C²  weights, 3 ReLUs
                      → ~45% fewer params AND 3× the nonlinearity for the same receptive field

Why it exists. It showed that depth with tiny filters beats shallow nets with big filters — more layers = more nonlinearity and a deeper feature hierarchy, while small kernels keep the parameter count and compute per layer manageable. Its uniform design made it a clean, reproducible baseline and a favorite feature extractor / backbone.

Where you see it (Qualcomm). Less deployed on-device today (VGG is parameter-heavy — ~138M params, ~500MB — far too big for a phone), but the 3×3-stack principle is everywhere in efficient nets, and VGG-style features are still used as perceptual losses for super-resolution/denoise (a camera-relevant use).

Answer. "VGG is the simplicity-and-uniformity CNN: a deep stack of only 3×3 convs and 2×2 max-pools, doubling channels after each pool, then FC layers and softmax — VGG-16 and VGG-19. The key idea is that stacking small 3×3 convs gives the same receptive field as a larger filter but with fewer parameters and more nonlinearity: two 3×3s equal a 5×5's receptive field, three equal a 7×7, at ~45% fewer weights and triple the ReLUs. It proved depth-with-small-filters works. Its drawback is huge parameter count — ~138M — so it's too heavy for mobile, but the 3×3-stack design lives on and its features are popular for perceptual losses."

Follow-ups / gotchas. The FC layers hold most of VGG's parameters (the conv part is comparatively small). VGG has no batch norm in the original (BN came later). It's the classic example of "deeper + smaller filters." Contrast with Inception (multi-scale) and ResNet (skip connections), which solve VGG's depth-scaling limits.

Seen in: Report 11 ("architectures of Inception, VGG"). Standard CNN-architecture expectation.

D2 · Q: Describe the Inception architecture. What problem do the 1×1 bottlenecks solve?¶

Frequency: 🔥🔥 Common (~2 reports) — explicitly asked in the ML & System Engineer loop ("architectures of Inception, VGG").

Concept — the basis. Inception / GoogLeNet (2014) replaces "pick one filter size" with multi-scale parallelism: an Inception module runs 1×1, 3×3, 5×5 convolutions and a 3×3 max-pool in parallel on the same input and concatenates their outputs along the channel axis — so the network sees features at several scales at once and learns which to use. The catch: doing 3×3 and 5×5 directly on a deep input is expensive, so each is preceded by a 1×1 conv bottleneck that cuts channel depth first (C2).

Why it exists. Two problems solved at once: (1) what filter size? — don't choose; compute several and let the data decide (multi-scale features); (2) cost — naive multi-scale is far too expensive, so the 1×1 bottlenecks make it affordable (the ~10× FLOP saving of C2). GoogLeNet hit VGG-class accuracy with ~20× fewer parameters (~5–7M vs ~138M) — a huge efficiency win (the original paper's headline figure is ~12× fewer than AlexNet). It also used global average pooling instead of giant FC layers and auxiliary classifiers during training to help gradient flow.

Where you see it (Qualcomm). The efficiency philosophy — multi-scale features at low cost via 1×1 bottlenecks — is exactly what mobile/NPU deployment wants: high accuracy per MAC. Later EfficientNet/MobileNet refine the same "do more with fewer FLOPs" goal that Inception pioneered.

Answer. "Inception/GoogLeNet does multi-scale feature extraction in parallel: each Inception module runs 1×1, 3×3, and 5×5 convs plus a 3×3 max-pool on the same input and concatenates the results channel-wise, so the net captures features at multiple scales and learns which matter. To make the 3×3 and 5×5 affordable, each is preceded by a 1×1 conv that reduces channel depth — the bottleneck — cutting FLOPs roughly 10×. The result matched VGG accuracy with roughly 20× fewer parameters (about 12× fewer than AlexNet, the paper's headline number), and it replaced big FC layers with global average pooling. The 1×1 bottleneck is the trick that makes multi-scale practical."

Follow-ups / gotchas. The 1×1 bottleneck is the single most important efficiency element (C2). Later versions (Inception-v2/v3) factorize 5×5 into two 3×3s and use asymmetric (1×7, 7×1) convs for further savings; Inception-v4/Inception-ResNet add residual connections. Auxiliary classifiers fight vanishing gradients (E4) in the original deep net.

Seen in: Report 11 ("architectures of Inception, VGG"). Standard CNN-architecture expectation.

D3 · Q: Explain ResNet. How do residual/skip connections fix vanishing gradients, and what are the advantages?¶

Frequency: 🔥🔥 Common (~2 reports) — explicitly asked in the ML & System Engineer loop ("Explain the architecture of ResNet? Advantages of ResNet?").

Concept — the basis. ResNet (2015) lets you train networks 100–1000+ layers deep by adding a skip (shortcut) connection that bypasses each block and adds the block's input back to its output:

y = F(x) + x          (F = the stacked conv→BN→ReLU layers; x = identity shortcut)

Why it fixes vanishing gradients (the key derivation). Differentiate the skip:

y = F(x) + x   ⟹   ∂y/∂x = ∂F/∂x + 1

Advantages (say all of these). (1) Trains very deep nets (50/101/152 layers) that plain nets can't — plain nets actually got worse with depth (the "degradation problem"), which residuals cure. (2) Solves vanishing gradients via the identity path. (3) Easier optimization — learning a residual near zero is easier than a full mapping, so adding layers never hurts (worst case the block learns identity). (4) Better accuracy at greater depth; ResNet-50 is the default vision backbone. (5) Bottleneck blocks (1×1 → 3×3 → 1×1) keep deep ResNets efficient (C2).

Where you see it (Qualcomm). ResNet-50 is the go-to backbone for camera/CV tasks (detection, segmentation, classification) and a standard quantization/benchmark target for the Hexagon NPU. The residual structure also quantizes reasonably and is the template for countless mobile backbones.

Answer. "ResNet adds a skip connection that carries the block input straight to its output: y = F(x) + x, so the block learns a residual F(x) = H(x) − x instead of the full mapping. That fixes vanishing gradients because of the identity path: differentiating gives ∂y/∂x = ∂F/∂x + 1, and that '+1' lets gradients flow backward multiplied by one rather than shrinking through a long chain of small factors — so 100-plus-layer nets stay trainable. Advantages: it cures the degradation problem where plain deep nets get worse, solves vanishing gradients, makes optimization easier since a block can fall back to identity, and delivers higher accuracy at great depth. ResNet-50 with 1×1 bottleneck blocks is the standard vision backbone."

Follow-ups / gotchas. When F(x) and x differ in shape (channel/stride change), the shortcut uses a 1×1 conv to match dimensions (the "projection shortcut"). The add is before the final ReLU. ResNet doesn't eliminate depth limits but pushes them far out. DenseNet concatenates instead of adds. Contrast the three families: VGG = depth + 3×3 simplicity; Inception = multi-scale + 1×1 bottlenecks; ResNet = skip connections for trainable depth.

Seen in: Report 11 ("Explain the architecture of ResNet? Advantages of ResNet?"). Standard CNN-architecture expectation.

E. Generalization & evaluation¶

E1 · Q: What is overfitting, and how do you prevent it (L2, dropout, etc.)?¶

Frequency: 🔥🔥 Common (~2–3 reports) — central to CV "real-world scenarios" (Report 5) and the training pipeline (Report 21).

Concept — the basis. Overfitting is when a model learns the training data's noise and idiosyncrasies instead of the generalizable pattern — it scores high on training data but poorly on unseen data (the train/val gap widens). The standard fixes (regularization = anything that improves test generalization, usually by constraining model complexity):

L2 gradient: adding (λ/2)Σwᵢ² to L adds λw to each gradient, so the update becomes w ← w − η(∂L/∂w + λw) = (1−ηλ)w − η∂L/∂w — i.e. weights decay toward zero each step (hence "weight decay").

Why it exists. A model with enough capacity can memorize the training set (zero train error) yet generalize terribly. Regularization expresses a prior — "prefer simpler explanations" — biasing the model toward functions that generalize. It directly trades a little training fit for much better test performance (the bias–variance trade-off, E2).

Where you see it (Qualcomm). A camera/audio model that's great in the lab but fails on real, varied field data is overfit; augmentation (noise/reverb/SNR-mixing for audio, color/geometry jitter for images) is the heavily-used fix (Report 21). Smaller, regularized models also fit the NPU's memory/compute budget.

Answer. "Overfitting is when the model memorizes training-set noise instead of the real pattern — low training error but high test error, with a growing train/val gap. I prevent it with, in rough priority: more and augmented data; L2 weight decay, which shrinks weights toward zero for a smoother function; dropout, which randomly zeros activations so neurons can't co-adapt and acts like an ensemble; early stopping on validation loss; batch norm's mild regularization; and reducing model capacity. L1 instead drives weights to exactly zero for sparsity. I detect it by watching the train-vs-validation gap."

Solution / good example — adding regularization in PyTorch:

# L2 weight decay (built into the optimizer):
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)

# Dropout layer (active in train(), automatically disabled in eval()):
self.drop = nn.Dropout(p=0.5)          # in __init__
x = self.drop(F.relu(self.fc(x)))      # in forward

# Early stopping (sketch): track best val loss, stop after N epochs of no improvement.

Follow-ups / gotchas. Dropout is off at inference (model.eval()); activations are scaled to compensate (inverted dropout). L2 = "weight decay" = Gaussian prior; L1 = Laplace prior → sparsity. Underfitting is the opposite (model too simple — high train and test error). Augment only training data. Don't rely on test-set feedback to tune (leakage).

Seen in: Report 5 ("optimization, evaluation"), Report 21 (augmentation as overfitting control). Standard ML expectation.

E2 · Q: Explain the bias–variance trade-off.¶

Frequency: 🔥 Occasional (~1–2 reports) — part of the "model selection / real-world scenarios" probe (Reports 5, 10).

Concept — the basis. A model's expected test error decomposes into three parts:

Error = Bias² + Variance + Irreducible noise

Visual mnemonic: high bias = consistently off-target (tight but wrong cluster); high variance = scattered all around the target (right on average, individually wild).

Why it exists. It's the fundamental tension in supervised learning: a model must be flexible enough to fit the real signal (low bias) but constrained enough to ignore noise (low variance). Every regularization choice (E1) and capacity decision is navigating this trade-off.

Where you see it (Qualcomm). Choosing model size for a camera/audio task: too small → it can't tell the classes apart (bias); too big for the available data / NPU budget → it overfits and is slow (variance + cost). Picking the model that generalizes and fits the device is this trade-off in practice.

Answer. "Total test error splits into bias squared, variance, and irreducible noise. Bias is error from a model too simple to capture the pattern — underfitting, high error on both train and test. Variance is error from a model so flexible it fits the training noise — overfitting, low train error but high test error. They trade off: more capacity lowers bias but raises variance. The aim is the sweet spot minimizing total error — found via the right model size plus regularization and validation. High bias looks like consistently wrong; high variance looks like scattered, dataset-dependent predictions."

Follow-ups / gotchas. More data reduces variance without adding bias (the cheapest lever). Regularization (E1) trades a little bias for less variance. The classic U-shaped test-error curve (though "double descent" complicates it for very large modern nets). Ensembling/bagging reduces variance; boosting reduces bias.

Seen in: Reports 5, 10 ("model selection"). Standard ML expectation.

E3 · Q: Define precision, recall, F1, and the confusion matrix. When does accuracy mislead?¶

Frequency: 🔥 Occasional (~1–2 reports) — implicit in CV "evaluation" (Report 5) and the wake-word false-positive question (Report 21).

Concept — the basis. For binary classification, the confusion matrix tallies the four outcomes:

                 Predicted +      Predicted −
   Actual +      TP (hit)         FN (miss)
   Actual −      FP (false alarm) TN (correct reject)

• Accuracy = (TP+TN)/total — fraction correct. Misleading on imbalanced data.
• Precision = TP/(TP+FP) — of the things I flagged positive, how many really are? (penalizes false alarms).
• Recall (sensitivity, TPR) = TP/(TP+FN) — of the actual positives, how many did I catch? (penalizes misses).
• F1 = 2·(P·R)/(P+R) — the harmonic mean of precision and recall; one number balancing both (harmonic mean punishes a low value, so F1 is high only when both are high).

Worked example (imbalanced): 1000 samples, 10 positives. A model that predicts "negative" for everything gets 99% accuracy but 0% recall — useless. Precision/recall/F1 expose that instantly.

Why it exists. Accuracy hides the type of error and collapses on class imbalance. Different applications care about different errors — a wake-word detector that false-alarms constantly (low precision) is annoying; one that misses the wake word (low recall) is broken. Precision/recall/F1 let you state and optimize the trade-off you actually care about.

Where you see it (Qualcomm). The Report 21 wake-word question — "how would you handle false positives ('Alex' vs 'Alexa')?" — is precisely a precision vs recall decision (raise the detection threshold → fewer false positives → higher precision, at some recall cost). Object detection uses mAP (precision–recall averaged over IoU thresholds). Camera face/scene detection lives on this trade-off.

Answer. "The confusion matrix counts true/false positives and negatives. Accuracy is the fraction correct but it lies on imbalanced data — predict the majority class and you look great while catching nothing. Precision is TP/(TP+FP): of what I flagged, how much is right — it penalizes false alarms. Recall is TP/(TP+FN): of the real positives, how many I caught — it penalizes misses. F1 is their harmonic mean, high only when both are. For a wake-word detector, raising the threshold trades recall for precision to cut false positives; detection uses mean Average Precision over IoU thresholds."

Follow-ups / gotchas. Precision/recall trade off via the decision threshold — sweep it to get the PR curve (or ROC/AUC). For multi-class, average per-class (macro/micro/weighted). F1 is the special case (β=1) of Fβ, which weights recall over precision. Detection adds IoU to decide what counts as a TP.

Seen in: Report 21 ("wake-word detection false positives"), Report 5 ("evaluation"). Standard ML expectation.

E4 · Q: What are vanishing and exploding gradients? How do you fix them?¶

Frequency: 🔥🔥 Common (~2 reports) — the reason behind ReLU/ResNet/BN questions (Reports 11, 21), so it's probed directly and indirectly.

Concept — the basis. In backprop, the gradient to an early layer is a product of many per-layer Jacobians (Wˡ and σ′(zˡ) factors — A2). Through a deep net: - Vanishing gradients: if those factors are consistently < 1 (e.g. saturating sigmoid/tanh where σ′ ≤ 0.25, or small weights), the product shrinks geometrically → early layers get ~0 gradient → they barely train. The deeper the net, the worse. - Exploding gradients: if the factors are consistently > 1 (large weights), the product grows geometrically → huge updates, oscillation, NaN loss. Common in RNNs over long sequences.

Intuition: 0.5²⁰ ≈ 1e-6 (vanished); 1.5²⁰ ≈ 3300 (exploded).

The fixes (a menu): | Problem | Fixes | |---|---| | Vanishing | ReLU (σ′=1 for z>0, B3); ResNet skip connections (+1 path, D3); batch norm (C4); good init (He/Xavier); LSTM/GRU gates (for RNNs) | | Exploding | gradient clipping (cap the norm); weight regularization; smaller LR; proper init; batch norm |

Why it exists. It's a direct consequence of the chain rule on deep computation graphs — repeated multiplication of many factors is numerically unstable unless those factors stay near 1. Recognizing it is the reason the field invented ReLU, careful initialization, normalization, and residual connections. It's the unifying "why" behind half of B/C/D.

Where you see it (Qualcomm). Why a deep camera/CV net won't converge or produces NaNs in training; why mobile backbones use ReLU + residual + BN (all three are vanishing-gradient cures); why RNN-based audio models need clipping or get replaced by transformers (G2).

Answer. "Backprop multiplies many per-layer factors together, so if they're consistently below 1 the gradient shrinks geometrically and early layers stop learning — vanishing gradients, classic with saturating sigmoid/tanh in deep nets. If they're consistently above 1, the gradient blows up to NaN — exploding gradients, common in RNNs. Fixes for vanishing: ReLU, whose derivative is 1 on the positive side; ResNet skip connections, which add a gradient path multiplied by exactly one; batch norm; and good initialization. For exploding: gradient clipping, smaller learning rate, and regularization. ReLU, residuals, and BN are basically the standard anti-vanishing toolkit."

Follow-ups / gotchas. Sigmoid's max derivative is 0.25 → vanishes fast in depth. He init for ReLU, Xavier/Glorot for tanh. ResNet's +1 is the cleanest structural fix (D3). LSTMs added gates precisely to keep gradients flowing over long sequences. Gradient clipping is a one-line RNN staple.

Seen in: Reports 11 (ResNet/why), 21 (LSTM vs Transformer, CNN training). Standard deep-learning expectation.

F. Computer vision & signal processing for ML¶

F1 · Q: Contrast the classic computer-vision pipeline with the deep-learning one.¶

Frequency: 🔥🔥 Common (~3 reports) — the CV-internship discussion (Report 3: edge/face detection, masking, segmentation, "general image-processing workflow") and CV-engineer round (Report 5).

Concept — the basis. Two eras of solving a vision task (say, object recognition):

Classic CV (hand-engineered):

image → preprocess (denoise, normalize)
      → hand-crafted features (edges via Sobel/Canny, corners via Harris,
        descriptors SIFT/SURF/HOG, color histograms, masking/morphology)
      → a classical ML classifier (SVM, random forest, k-NN)
      → output

Deep CV (learned end-to-end):

image → CNN (conv/pool/BN/ReLU stacks) that LEARNS the features → task head → output

Why the shift happened. Hand-crafted features are brittle (tuned per dataset, fail under lighting/viewpoint/occlusion changes) and labor-intensive. CNNs learn features optimized for the task and generalize far better given enough data and compute — the 2012 AlexNet result triggered the switch. Trade-offs: classic CV is interpretable, data-light, and cheap (great when data is scarce or the rule is simple — e.g. a fixed fiducial marker); deep CV is accurate and general but data- and compute-hungry.

Where you see it (Qualcomm). Both coexist: the ISP does classic, deterministic image processing (demosaic, denoise, tone-map — see 05_camera_isp_multimedia.md), while on top a CNN on the NPU does scene detection, segmentation (portrait/bokeh), face/object detection, super-resolution. Classic ops (edge detection, masking) still appear as cheap pre/post-processing. Knowing both lets you pick the cheapest tool that meets the quality bar — a real mobile-power concern.

Answer. "Classic CV is a hand-engineered pipeline: preprocess, then extract designed features — edges with Sobel/Canny, descriptors like SIFT/HOG, color histograms, morphology — then feed a classical classifier like an SVM. Deep CV replaces the hand-crafted features with a CNN that learns the whole feature hierarchy end-to-end from data. The field switched because hand-crafted features are brittle and labor-intensive, while CNNs generalize much better given data and compute. But classic CV is still valuable when it's interpretable, data is scarce, or the task is simple and cheap. On a phone both coexist — the ISP does deterministic image processing, and CNNs on the NPU handle the learned vision tasks."

Follow-ups / gotchas. Classic features (SIFT/ORB) still win for geometric tasks (SLAM, stereo matching, calibration) where precision and no-training-data matter. Deep models need labeled data + compute. Hybrid pipelines are common. Edge/face detection: classic = Canny/Viola–Jones; deep = CNN detectors (F2).

Seen in: Report 3 ("edge detection, face detection, image segmentation, general image-processing workflow"), Report 5 (CV architectures). Standard CV expectation. Camera ISP specifics → 05_camera_isp_multimedia.md.

F2 · Q: Explain object-detection architectures (R-CNN, YOLO). What's the trade-off?¶

Frequency: 🔥 Occasional (~1–2 reports) — explicitly asked in the CV-engineer loop ("CNNs, R-CNN, YOLO and various object detection architectures").

Concept — the basis. Detection = classify + localize (draw boxes). Two families:

• Two-stage (R-CNN family): first propose candidate regions, then classify each.
• R-CNN → run a CNN on each of ~2000 region proposals (slow).
• Fast R-CNN → run the CNN once on the whole image, pool features per region (RoI pooling).
• Faster R-CNN → a learned Region Proposal Network generates proposals — fully end-to-end. Accurate, slower.
• One-stage (YOLO / SSD): no separate proposal step — a single CNN pass predicts boxes + class probabilities directly on a grid. YOLO ("You Only Look Once") regresses boxes for every grid cell in one shot. Fast (real-time), slightly less accurate (historically; modern YOLO closes much of the gap).

Why both exist. The fundamental speed–accuracy trade-off. Two-stage refines proposals → higher precision, especially for small/overlapping objects, but more compute. One-stage does it in a single pass → real-time throughput at some accuracy cost. You pick based on the latency budget.

Where you see it (Qualcomm). Real-time, on-device detection (faces, objects, hands for AR/XR, scene elements) demands one-stage detectors — YOLO/SSD-style — because they hit the frame-rate and power budget on the Hexagon NPU. Quantized YOLO (INT8) on the NPU is a canonical mobile-CV deployment (G3).

Answer. "Detection both classifies and localizes objects with bounding boxes. Two-stage detectors — the R-CNN family up to Faster R-CNN — first propose regions, then classify them; Faster R-CNN uses a learned region-proposal network and is accurate but heavier. One-stage detectors like YOLO and SSD skip proposals and predict boxes and classes in a single CNN pass over a grid — much faster, historically a bit less accurate. It's a speed-accuracy trade-off. On a phone we use one-stage detectors like YOLO, quantized to INT8 on the NPU, because they meet the real-time latency and power budget."

Follow-ups / gotchas. Both use NMS (non-max suppression) to dedupe overlapping boxes and IoU + mAP for evaluation (E3). Anchor boxes (priors) vs anchor-free. Detection loss = classification (CE/focal) + box regression (smooth-L1/IoU). Segmentation (Mask R-CNN, U-Net) adds per-pixel masks.

Seen in: Report 5 ("CNNs, R-CNN, YOLO and various object detection architectures"). Standard CV expectation.

F3 · Q: What signal-processing basics do you need for ML? (How do you feed audio to a CNN?)¶

Frequency: 🔥🔥 Common (~4 reports for signal-processing broadly) — explicitly in the Audio-ML loop (Report 21: STFT→mel, CNN on spectrograms) and the DSP-heavy System-Engineer rounds (Reports 16, 30, 48, 49 — Fourier transform, digital filtering, detection & estimation).

Concept — the basis. A raw audio waveform is a 1-D time series; CNNs (designed for images) want a 2-D grid. The bridge is time-frequency analysis:

waveform → frame (e.g. 25 ms windows, 10 ms hop) + window function (Hann)
         → STFT (FFT per frame) → magnitude spectrogram  (time × frequency)
         → mel filter bank (matrix multiply, ~40–80 mel bins)  → mel-spectrogram
         → log → (optionally) MFCC via DCT
         → feed the 2-D spectrogram into a CNN as an "image"

Why a CNN works on a spectrogram: local time-frequency patterns (a formant, a phoneme onset, a wake-word's spectral signature) are exactly the kind of local, translation-equivariant features a conv kernel detects — so the audio problem becomes an image problem.

Why it exists. The raw waveform is high-rate and the information humans/models care about lives in the frequency content over time, not the raw samples. STFT/mel turns audio into a compact, perceptually-meaningful 2-D representation that downstream ML (CNNs, transformers) handles well. The same DSP underlies the modem/wireless and detection-estimation questions Qualcomm asks (filtering, FFT, Fourier).

Where you see it (Qualcomm). Exactly Report 21: "CNN on spectrogram inputs for Alexa wake-word recordings"; "audio processing from STFT to mel-scale via filter-bank multiplication." Qualcomm's always-on, low-power audio (Sensing Hub / Hexagon) runs wake-word and keyword spotting on mel-spectrogram CNNs. The signal-processing fundamentals (Fourier, filtering, sampling) also pervade the modem/RF and DSP roles (Reports 16, 30, 38, 48, 49).

Answer. "Audio is a 1-D waveform, but a CNN wants a 2-D grid, so I convert it with time-frequency analysis: frame the signal into short overlapping windows, take an FFT per frame — that's the STFT — to get a spectrogram of how frequencies evolve over time, then apply a mel filter bank (a matrix multiply) to get a perceptually-spaced mel-spectrogram, take the log, and optionally a DCT for MFCCs. That 2-D mel-spectrogram feeds the CNN like an image — and it works because local time-frequency patterns like a wake-word's signature are exactly the local features convolution detects. Core DSP behind it: Fourier transform / FFT, the mel scale, filtering, and Nyquist sampling. This is how a wake-word detector runs on the always-on audio DSP."

Follow-ups / gotchas. SpecAugment (mask time/frequency bands) augments spectrograms (B5). Convolution in time = multiplication in frequency (the convolution theorem). MFCC = DCT of log-mel (decorrelates, classic for speech). Window length trades time vs frequency resolution (uncertainty principle). 1-D convs can also work directly on waveforms (WaveNet). Detection & estimation (Reports 48, 49) = the statistical-signal-processing cousin (matched filters, MAP/ML estimators).

Seen in: Report 21 (STFT→mel, CNN on spectrograms, augmentation), Reports 16/30/38/48/49 (Fourier transform, digital filtering, detection & estimation, signal processing). Standard signal-processing-for-ML expectation.

G. Generative AI, transformers & on-device inference¶

G1 · Q: Explain the transformer and the attention mechanism.¶

Frequency: 🔥🔥 Common (~3 reports) — explicitly in the Audio-ML loop (Report 21: Whisper, encoder-decoder cross-attention, seq2seq) and the GenAI senior loop (Report 25: "rapid-fire Generative AI concepts").

Concept — the basis. The transformer (2017, "Attention Is All You Need") replaced recurrence with self-attention: every token attends to every other token in parallel, so the model captures long-range dependencies without stepping through a sequence.

Scaled dot-product attention (the core formula — verified):

Attention(Q, K, V) = softmax( Q·Kᵀ / √dₖ ) · V

The rest of the architecture: multi-head attention (several attention "heads" learn different relationships in parallel), positional encodings (since attention is order-agnostic, inject position info), feed-forward sublayers, residual connections + LayerNorm (D3, C4), stacked into encoder and/or decoder blocks. Cross-attention in an encoder-decoder lets the decoder attend to the encoder's output (used in translation and in Whisper for speech-to-text). GPT-style models are decoder-only with causal/masked self-attention (each token sees only earlier tokens).

Why it exists. RNNs/LSTMs process sequences sequentially (slow, can't parallelize over time) and struggle with long-range dependencies (vanishing gradients over many steps, E4). Self-attention computes all pairwise interactions in parallel with a direct path between any two tokens — far better long-range modeling and massively more parallelizable on GPUs/NPUs. That scalability is what made today's large language and multimodal models possible.

Where you see it (Qualcomm). Transformers are the architecture of on-device generative AI — the whole point of the Hexagon NPU's recent generations (running LLMs/Whisper-style ASR on-device, no cloud round-trip). The Report 21 audio questions (Whisper = conv front-end + transformer encoder-decoder; seq2seq with cross-attention) and Report 25's GenAI loop are exactly this. Attention's compute pattern (big matmuls + softmax) is what the NPU is increasingly designed to accelerate.

Answer. "A transformer replaces recurrence with self-attention so every token attends to every other token in parallel. The core is scaled dot-product attention: project each token to a query, key, and value; score with Q·Kᵀ; divide by √dₖ to keep softmax gradients healthy; softmax to get weights summing to one; and multiply by V for a weighted blend. Multi-head attention runs several of these in parallel to capture different relationships, with positional encodings, feed-forward layers, and residual+LayerNorm around them, stacked into encoder/decoder blocks. It beat RNNs because it parallelizes over the sequence and gives a direct path between distant tokens, fixing long-range dependencies. Whisper, for example, is a conv front-end plus a transformer encoder-decoder with cross-attention. This is the architecture behind on-device generative AI on the NPU."

Follow-ups / gotchas. Self-attention is O(n²) in sequence length — the motivation for efficient/sparse/flash attention (a real on-device concern). Causal mask = decoder-only LMs. KV-cache speeds autoregressive decoding (and dominates inference memory — key for on-device LLMs). Positional info via sinusoidal or RoPE. LayerNorm (not BatchNorm) because sequence/batch stats are unstable.

Seen in: Report 21 (Whisper, encoder-decoder cross-attention, seq2seq), Report 25 (GenAI rapid-fire). Standard modern-DL expectation.

G2 · Q: Compare LSTM and Transformer.¶

Frequency: 🔥 Occasional (~1 report) — explicitly in the Audio-ML loop (Report 21: "Compare LSTM vs Transformer — computational requirements and parallelization").

Concept — the basis. Both model sequences; they differ fundamentally in how:

LSTM uses input/forget/output gates and a cell state to carry information across many steps and mitigate vanishing gradients — a big improvement over vanilla RNNs, but still sequential and limited at very long range. Transformers drop recurrence entirely for attention (G1).

Why it matters. The trade-off is parallelism + long-range power (transformer) vs streaming, low-memory, length-linear (LSTM). Transformers dominate when you can batch/parallelize and have memory; LSTMs/streaming models still appeal for always-on, low-latency, low-memory streaming inference.

Where you see it (Qualcomm). Directly the Report 21 question. On-device: transformers (Whisper-style) for high-accuracy ASR when the NPU can handle them; lightweight RNN/streaming or small-conv models for always-on, ultra-low-power keyword spotting where the O(n²) attention and KV-cache memory are too costly. The choice is an on-device compute/memory/latency decision.

Answer. "An LSTM is a recurrent net that processes a sequence step by step, using input/forget/output gates and a cell state to carry information and fight vanishing gradients. A transformer drops recurrence for self-attention, processing all tokens in parallel. So the transformer trains much faster — it parallelizes over the sequence — and models long-range dependencies far better via direct token-to-token attention, which is why it dominates. The costs are that attention is quadratic in sequence length and inference memory grows with context via the KV-cache, whereas an LSTM is linear in length with a small fixed state. On-device that trade-off matters: transformers for accuracy when the NPU can afford them, lightweight streaming/RNN models for always-on, low-memory keyword spotting."

Follow-ups / gotchas. GRU = simpler LSTM (2 gates). Transformers need positional encodings (LSTMs get order for free from recurrence). For streaming/causal audio, attention must be masked/windowed. The O(n²) cost drives efficient-attention research for long contexts on-device.

Seen in: Report 21 ("Compare LSTM vs Transformer — computational requirements and parallelization"). Standard sequence-modeling expectation.

G3 · Q: What is quantization, and why is it essential for on-device inference (the Hexagon NPU)?¶

Frequency: 🔥🔥 Common (~2–3 reports in spirit) — the Qualcomm-specific capstone behind every "deployment / optimization" question (Reports 5, 10, 21) and the on-device-AI focus of the role.

Concept — the basis. Quantization converts a model's weights/activations from 32-bit floating point (FP32) to low-bit integers — usually INT8 (and FP16/INT16 for sensitive parts). It maps a float range to integers with a scale and zero-point:

real ≈ scale · (q − zero_point)        q is an 8-bit integer

Two flavors: - Post-training quantization (PTQ): quantize an already-trained FP32 model, using a small calibration set to pick per-tensor/per-channel ranges. Fast, no retraining; small accuracy drop. - Quantization-aware training (QAT): simulate quantization (fake-quant ops) during training so the model learns to be robust to it. More work, best accuracy — used when PTQ loses too much.

Why it exists. A phone has tight power, thermal, memory, and latency limits and no cloud round-trip for private/real-time inference. FP32 is too big/slow/power-hungry. Integer arithmetic on a dedicated tensor accelerator delivers the throughput-per-watt that real-time, always-on, on-device AI needs. Quantization is the bridge from a trained float model to something that actually runs on the NPU.

Where you see it (Qualcomm). This is the center of gravity of Qualcomm's AI story: the Qualcomm AI Engine = Hexagon NPU (scalar + vector HVX + tensor accelerator) + the Adreno GPU + Kryo CPU, with a heterogeneous runtime (QNN / SNPE / AI Engine Direct, AIMET for quantization). You quantize a YOLO/ResNet/Whisper to INT8, deploy via QNN, and it runs entirely on-device — low latency, low power, data never leaves the phone. Camera computational photography (night mode, segmentation), always-on audio, and on-device GenAI all ride this path.

Answer. "Quantization converts weights and activations from FP32 to low-bit integers, typically INT8, via a scale and zero-point. It makes the model about 4× smaller and 2–4× faster at much lower power because integer MACs are cheap — and the Hexagon NPU is built for it, favoring INT8 weights with 16-bit activations. Two approaches: post-training quantization, which just calibrates ranges on a trained model — fast, small accuracy hit; and quantization-aware training, which simulates quantization during training for the best accuracy when PTQ drops too much. It's essential on-device because a phone has tight power, thermal, memory, and latency limits and wants no cloud round-trip — so we quantize the model and run it on the Qualcomm AI Engine, the Hexagon NPU with its tensor accelerator, entirely on the device. That's how real-time camera CV, always-on audio, and on-device GenAI work."

Follow-ups / gotchas. Per-channel quantization (separate scale per output channel) preserves accuracy better than per-tensor. Fold BN into conv before quantizing (C4). ReLU6 caps activation range for clean quantization (B3). Outliers in activations hurt — clipping/calibration matters. Pruning + quantization + knowledge distillation are the mobile-compression trio. Tools: AIMET (Qualcomm's quantization toolkit), QNN/SNPE runtime. Deeper SoC/number-format detail → 09_computer_arch_digital_design.md.

Seen in: Reports 5, 10, 21 ("deployment, optimization" — the on-device endpoint of every ML pipeline). Standard on-device-inference expectation; the Qualcomm-specific high-value topic.

§ Encyclopedia — searchable glossary¶

Every bold-italic term used above is defined here, alphabetically, with why it exists / where you see it and a micro-example. Use your editor's find (⌘F / Ctrl-F) to jump to a term.

Activation function — the per-neuron nonlinearity a = σ(z) (B3). Why: without it, stacked layers collapse to one linear map. Where: ReLU in CNN hidden layers, softmax at a classifier's output. relu(z)=max(0,z).

Adam — adaptive optimizer tracking first + second gradient moments with bias correction (B2). Why: per-parameter adaptive step + momentum → fast, robust, little tuning. Where: default for training most deep nets. θ−=η·m̂/(√v̂+ε), lr 1e-3, betas (0.9,0.999).

Attention (scaled dot-product) — softmax(QKᵀ/√dₖ)V; each token weights all others (G1). Why: parallel, direct long-range token interactions. Where: the core of transformers; Whisper, GPT, on-device GenAI.

Augmentation — label-preserving input perturbations (flip/crop/noise; audio noise/reverb/SpecAugment) (B5, E1). Why: cheaply multiplies data → fights overfitting. Where: every training pipeline; MUSAN/SNR-mixing for wake-word audio.

Backpropagation — chain rule run backward over the computation graph to get all gradients in ~one backward pass (A2). Why: makes training deep nets tractable. Where: loss.backward(); the theory behind every trained model. δˡ=(Wˡ⁺¹)ᵀδˡ⁺¹⊙σ′(zˡ).

Batch normalization (BN) — normalize pre-activations per batch, then learnable scale/shift (C4). Why: smooths the loss landscape → higher LR, faster, robust to init. Where: conv→BN→ReLU blocks; folded into conv at inference.

Bias (neuron) — the additive term b in z=w·x+b (A1). Why: shifts the decision boundary off the origin. Not the same as statistical bias (E2).

Bias–variance trade-off — total error = bias² + variance + noise (E2). Why: the core tension between underfitting (bias) and overfitting (variance). Where: choosing model capacity + regularization.

Confusion matrix — TP/FP/FN/TN tally for a classifier (E3). Why: exposes which errors, unlike accuracy. Where: every classification eval; basis of precision/recall.

Convolution — sliding-kernel dot product over a feature map; parameter-sharing local operator (C1). Why: locality + weight sharing + translation equivariance → efficient on images. Where: every CNN; dominant NPU compute. out=(W−K+2P)/S+1.

Cross-attention — attention where queries come from one sequence, keys/values from another (G1). Why: lets a decoder attend to an encoder's output. Where: translation, Whisper ASR, seq2seq.

Cross-entropy — classification loss −Σ yᵢ log ŷᵢ (B4). Why: negative log-likelihood of the truth; with softmax the gradient is the clean ŷ−y (no saturation stall). Where: every classifier; wake-word detector.

Dead/dying ReLU — a neuron stuck outputting 0 with 0 gradient forever (B3). Why it happens: a big update/bias pushes it permanently negative. Fix: Leaky ReLU, lower LR, careful init.

Dropout — randomly zero a fraction p of activations each training step (E1). Why: prevents co-adaptation → acts like an ensemble → regularizes. Where: FC layers; off at inference (model.eval()).

Epoch — one full pass over the training set (B1). Note: one epoch = many mini-batch steps. Where: training loops; early-stopping is counted in epochs.

Exploding gradients — gradient product grows geometrically → NaN (E4). Fix: gradient clipping, smaller LR, good init. Where: RNNs over long sequences.

F1 score — harmonic mean of precision and recall, 2PR/(P+R) (E3). Why: one number, high only when both are high. Where: imbalanced classification, detection screening.

Feature map — the output volume (H×W×C) of a conv layer (C1). Where: every CNN stage; pooling/stride shrink it, channels grow.

FLOPs / MACs — floating-point ops / multiply-accumulates; the compute cost metric (C2). Why: proxy for latency/energy on the NPU. Example: 5×5,192→32 on 28×28 ≈ 120M MACs.

Forward pass — compute outputs layer by layer, aˡ=σ(Wˡaˡ⁻¹+bˡ) (A1). Where: inference is the forward pass; the only thing that runs on the NPU.

Fourier transform / FFT — decompose a signal into frequency components (F3). Why: the info in audio/RF lives in frequency. Where: STFT, filtering, modem/DSP roles.

Gradient descent — iterative minimization θ←θ−η∇L (B1). Variants: batch (all data), SGD (one sample), mini-batch (the default). Where: how every net is trained.

Inception / GoogLeNet — multi-scale module (parallel 1×1/3×3/5×5/pool, concatenated) with 1×1 bottlenecks (D2). Why: multi-scale features cheaply → VGG accuracy at ~20× fewer params (~12× fewer than AlexNet). Where: efficiency-CNN lineage.

Learning rate (η) — step-size multiplier in gradient descent (B1). Why: too small = slow/stall, too large = overshoot/diverge. Where: the single most important hyperparameter; usually scheduled.

Loss function — the scalar minimized in training (B4). Why: defines "what wrong means." Where: MSE for regression, cross-entropy for classification.

LSTM — gated RNN (input/forget/output gates + cell state) for sequences (G2). Why: gates carry info across steps, mitigating vanishing gradients. Where: streaming/low-memory sequence tasks; largely superseded by transformers.

Max pooling — take the max in each window, downsample, no weights (C3). Why: keep salient features + small shift-invariance + cheaper. Backprop: gradient routes only to the argmax. Contrast: average/global-average pooling.

Mel spectrogram — STFT magnitude mapped through a perceptual mel filter bank (F3). Why: compact, hearing-aligned 2-D audio rep for CNNs. Where: wake-word/ASR front-ends on the audio DSP.

Mini-batch — a small subset (e.g. 32–256) used per gradient step (B1). Why: stable gradient + vectorizes on GPU/NPU + helpful noise. Where: the practical training default.

Momentum — velocity term smoothing SGD updates (B2). Why: damps oscillation, accelerates consistent directions. Where: SGD-momentum; the first moment in Adam.

MSE (mean squared error) — regression loss (1/N)Σ(ŷ−y)² (B4). Why: natural distance for continuous targets. Where: super-resolution, depth, box-coordinate regression. Pitfall: saturating gradients if used with sigmoid for classification.

Neuron / perceptron — a=σ(w·x+b); perceptron = one neuron with a threshold (A1). Why: smallest learnable-linear + fixed-nonlinear unit. Where: every layer of every net.

Overfitting — model memorizes training noise; low train, high test error (E1). Fix: more/augmented data, L2/L1, dropout, early stop, less capacity. Detect: train-vs-val gap.

Padding — border zeros added before convolution (C1). Why: "same" padding preserves spatial size; "valid" shrinks it. Where: controlling feature-map dimensions through a CNN.

Pooling — weightless downsampling that summarizes windows (C3). Why: reduce compute, grow receptive field, add shift-invariance. Where: classic CNN backbones; global-avg-pool replaces FC heads.

Precision — TP/(TP+FP); fraction of positive predictions that are correct (E3). Why: penalizes false alarms. Where: raise the threshold to boost it (fewer wake-word false positives).

Quantization — FP32 → INT8 (scale + zero-point) for size/speed/power (G3). Why: enables real-time, low-power on-device inference. Where: QNN/AIMET → Hexagon NPU; PTQ vs QAT.

Recall (sensitivity, TPR) — TP/(TP+FN); fraction of actual positives caught (E3). Why: penalizes misses. Where: a detector that misses the wake word has low recall.

Receptive field — the input region influencing one output activation (C1). Why: grows with depth → CNN sees edges→objects. Where: the "two 3×3s = one 5×5" VGG insight.

Regularization — anything that improves test generalization by constraining complexity (E1). Examples: L2 weight decay, dropout, early stopping, augmentation, BN.

ReLU — max(0,z), derivative 1 for z>0 (B3). Why: non-saturating gradient + cheap → trainable deep CNNs. Where: the CNN default; ReLU6 quantizes cleanly on the NPU. Pitfall: dying ReLU.

ResNet — CNN with identity skip connections y=F(x)+x (D3). Why: the +1 in ∂y/∂x gives gradients a direct path → trains 100+ layers; cures the degradation problem. Where: ResNet-50, the default vision backbone.

Sigmoid — 1/(1+e⁻ᶻ), range (0,1) (B3). Why: smooth probability-like output. Pitfall: saturates → vanishing gradients in deep nets; use for binary output/gates, not deep hidden layers.

Softmax — eᶻⁱ/Σeᶻʲ, turns logits into a probability distribution (A1, B4). Why: multi-class output that sums to 1. Where: classifier output layer; pairs with cross-entropy; inside attention.

STFT — short-time Fourier transform: FFT over short sliding windows (F3). Why: shows how frequencies evolve over time. Where: the spectrogram step feeding audio CNNs.

Stride — kernel step size in convolution/pooling (C1). Why: stride 2 downsamples ~2× (cheaper, smaller output). Where: resolution/compute control; stride-2 conv as a pooling alternative.

Tanh — (eᶻ−e⁻ᶻ)/(eᶻ+e⁻ᶻ), range (−1,1) (B3). Why: zero-centered sigmoid. Pitfall: still saturates. Where: LSTM cell, older hidden layers.

Transformer — attention-based sequence model, no recurrence (G1). Why: parallel + strong long-range modeling → scales to LLMs. Where: Whisper, GPT, on-device GenAI on the NPU. Cost: O(n²) attention.

Vanishing gradients — gradient product shrinks geometrically → early layers stop learning (E4). Cause: saturating activations / small weights through depth. Fix: ReLU, ResNet skips, BN, good init.

VGG — deep, uniform CNN of stacked 3×3 convs + 2×2 pools (D1). Why: depth with small filters beats shallow + big filters; fewer params + more nonlinearity per receptive field. Pitfall: ~138M params, too heavy for mobile; lives on as a perceptual-loss backbone.

Weight decay (L2) — penalty λΣwᵢ² that shrinks weights each step (E1). Why: smoother, simpler function → less overfitting. Where: weight_decay= in the optimizer; AdamW decouples it.

Weight sharing — one kernel reused across all spatial positions in a conv (C1, A3). Why: drastically fewer parameters + translation equivariance. Consequence: the kernel gradient sums over all positions in backprop.

YOLO — one-stage real-time object detector (single pass predicts boxes + classes) (F2). Why: speed for real-time/on-device. Where: quantized INT8 YOLO on the Hexagon NPU. Contrast: two-stage Faster R-CNN (more accurate, slower).

Hexagon NPU / Qualcomm AI Engine — Qualcomm's on-device AI accelerator: scalar + vector (HVX) + tensor units, run via QNN/SNPE (G3). Why: high throughput-per-watt integer inference. Where: runs quantized camera CV, audio, and GenAI models entirely on-device.

im2col — reshaping conv patches into columns so convolution becomes a matrix multiply (A4, C1). Why: maps conv onto fast MAC/tensor hardware. Where: NPU/GPU conv implementations.

Zero-point / scale — the integer-mapping parameters in quantization, real≈scale·(q−zero_point) (G3). Why: represent a float range in 8 bits. Where: per-tensor or per-channel during PTQ/QAT.

§ Last-5-minutes cheat sheet¶

• Neuron: a = σ(w·x + b). Forward pass = matmul → add bias → activate, layer by layer. No nonlinearity ⇒ collapses to one linear map.
• Backprop = chain rule backward; error signal δˡ = (Wˡ⁺¹)ᵀδˡ⁺¹ ⊙ σ′(zˡ); grads ∂L/∂W = δ·aᵀ_prev, ∂L/∂b = δ. With softmax+CE, output δ = ŷ − y.
• Conv backprop: kernel grad = input ⊛ upstream (summed over positions); input grad = full-conv with flipped kernel; max-pool routes grad to the argmax. Writing it in C = flat arrays + nested loops + memset grads to 0.
• GD: θ ← θ − η∇L. batch (all) · SGD (one) · mini-batch (default). η too big ⇒ diverge/NaN; too small ⇒ stall. Adam = momentum (1st moment) + per-param scaling (2nd moment) + bias correction; lr 1e-3, betas (0.9,0.999).
• Activations: sigmoid/tanh saturate ⇒ vanishing grads. ReLU max(0,z), σ′=1 ⇒ default; dying ReLU ⇒ Leaky ReLU. ReLU6 quantizes cleanly.
• Loss: MSE = regression; cross-entropy = classification (clean ŷ−y gradient, no saturation stall).
• Conv: out=(W−K+2P)/S+1; stride downsamples, "same" padding preserves size; receptive field grows with depth. 1×1 conv mixes channels → bottleneck → ~10× FLOP cut (28×28×192: 120M → 12.4M).
• BatchNorm: normalize per batch + learnable γ,β; smooths loss landscape ⇒ higher LR; fold into conv at inference.
• Architectures: VGG = 3×3 stacks (simple, heavy); Inception = multi-scale + 1×1 bottlenecks; ResNet = skip y=F(x)+x, ∂y/∂x=∂F/∂x+1 ⇒ fixes vanishing grads ⇒ 100+ layers.
• Overfitting ⇒ data/augmentation, L2 (weight decay), dropout, early stop. Bias = underfit; variance = overfit.
• Metrics: Precision TP/(TP+FP) (false alarms); Recall TP/(TP+FN) (misses); F1 = harmonic mean. Accuracy lies on imbalance.
• Vanishing/exploding = grad product <1 shrinks / >1 explodes ⇒ ReLU + ResNet + BN + clipping.
• Audio→CNN: waveform → STFT → mel-spectrogram → CNN (local time-freq features). Fourier/FFT/filtering = DSP basics.
• Transformer: softmax(QKᵀ/√dₖ)V; parallel, long-range; beats LSTM (sequential, O(n) vs O(n²)). Whisper = conv + transformer enc-dec.
• On-device: quantize FP32→INT8 (4× smaller, low power) via PTQ/QAT → run on Hexagon NPU / Qualcomm AI Engine (QNN/AIMET), no cloud.

Evidence base: qualcomm_camera_interview_experiences.md. Frequencies are approximate (sample = 75 reports, some via aggregator pages). Diagrams in assets/ (ml_*.svg). Cross-references: image-processing/ISP pipeline, 3A, HDR, denoise → 05_camera_isp_multimedia.md · DSA (linked lists/trees/bit-tricks) → 03_dsa.md · C memory/pointers behind the C-backprop answer → 01_c_programming.md · OS/threading for training parallelism → 04_os.md · SoC/accelerators/number formats/fixed-point → 09_computer_arch_digital_design.md · "explain your ML project" framing → 11_behavioral_hr_projects.md.