Wei-Cheng Chiu

邱偉誠

Taipei, Taiwan · UTC+8

making every FLOP count

Blog

FedAvg, DP-SGD & secure aggregation

On this page ⌄

1. 1. The FedAvg training loop｜FedAvg 訓練迴圈
2. 2. The Non-IID problem｜Non-IID 問題
3. 3. Differential privacy: local data can still leak｜Differential privacy：本地資料仍可能外洩
4. 4. Secure aggregation hides individual updates｜Secure aggregation 隱藏個別更新
5. 5. Implementation checks｜實作檢查

FedAvg Under Non-IID Data: What DP-SGD and Secure Aggregation Protect｜Non-IID 資料下的 FedAvg：DP-SGD 與 Secure Aggregation 各自保護什麼

2026-05-31 · updated 2026-07-21 · privacy-preserving ML

Federated learning with formal DP guarantees · 具有正式 DP 保證的聯邦學習 · Source: Google Research

The federated-learning-lab¹ implements FedAvg, FedProx, SCAFFOLD, DP-SGD, and secure aggregation from scratch. Its tests and experiments show where Non-IID data causes client drift, where extra communication rounds matter more than a different optimizer, and which privacy threat each defense covers.

federated-learning-lab¹ 從頭實作 FedAvg、FedProx、SCAFFOLD、DP-SGD 與 secure aggregation。測試與實驗顯示 Non-IID 資料如何造成 client drift、哪些情況下增加 communication rounds 比換 optimizer 更有影響，以及每種隱私防禦各自處理哪個威脅。

The repository includes 33/33 passing tests and cross-checks the implementations against the cited literature.

Repository 內含 33／33 通過的測試，並以引用文獻交叉檢查各項實作。

1. The FedAvg training loop｜FedAvg 訓練迴圈

Raw training examples remain on each client, while model updates are sent to an aggregation server. The canonical FedAvg loop is:²

原始訓練樣本留在各個 client，model updates 則傳給 aggregation server。標準 FedAvg 迴圈如下：²

1. The server broadcasts the global model.
   Server 廣播 global model。
2. Each client runs a few local SGD epochs on its own data.
   每個 client 用自己的資料執行數個 local SGD epochs。
3. Each client returns its updated weights.
   每個 client 回傳更新後的 weights。
4. The server computes a data-size-weighted average to produce the next global model.
   Server 依 client data size 加權平均，產生下一版 global model。

FedAvg keeps raw examples local, but heterogeneous client distributions make the local updates drift in different directions.

FedAvg 讓原始樣本留在本地，但異質的 client distributions 會使各地更新往不同方向偏移。

2. The Non-IID problem｜Non-IID 問題

FedAvg works best when participating clients sample similar distributions. Real clients differ: one hospital sees a different case mix from another, and one phone’s keyboard sees different language. Under Non-IID data, the average of those local updates can land in a region that is poor for many clients. This is client drift.

參與的 clients 取樣自相近分布時，FedAvg 最容易運作。現實中的 clients 並不相同：不同醫院會遇到不同病例組合，不同手機鍵盤也會看到不同語言。資料是 Non-IID 時，local updates 的平均可能落在許多 clients 都表現不佳的區域，這就是 client drift。

The lab implements and measures two common corrections:

Lab 實作並量測兩種常見修正方法：

• FedProx adds a proximal term that penalizes movement too far from the global model. It can stabilize training when clients are heterogeneous.³
  FedProx 加入 proximal term，懲罰離 global model 太遠的更新；clients 異質時可提高穩定性。³
• SCAFFOLD tracks control variates that estimate and subtract the drift direction. It communicates more state, but corrects bias that FedProx only damps.⁴
  SCAFFOLD 維護 control variates，估計並扣除 drift direction。它要傳輸更多狀態，但能修正 FedProx 只會抑制的 bias。⁴

On the label-skewed MNIST split in this repository, FedProx and SCAFFOLD did not consistently outperform FedAvg by a large margin. Increasing communication rounds sometimes had more effect.

在這個 repository 的 label-skewed MNIST split 上，FedProx 與 SCAFFOLD 並未持續大幅勝過 FedAvg；增加 communication rounds 有時反而更有影響。

3. Differential privacy: local data can still leak｜Differential privacy：本地資料仍可能外洩

Keeping raw data on-device does not by itself provide privacy. Model updates can expose information through membership inference or gradient inversion. DP-SGD adds a formal differential-privacy bound to training.⁵

把原始資料留在裝置上，本身並不構成隱私保證。Model updates 仍可能透過 membership inference 或 gradient inversion 洩漏資訊。DP-SGD 會在訓練中加入正式的 differential-privacy bound。⁵

DP-SGD changes each training step in two ways:

DP-SGD 對每個 training step 做兩項修改：

1. Per-sample gradient clipping bounds each example’s contribution to a maximum norm C.
   Per-sample gradient clipping 把每筆樣本的貢獻限制在最大 norm C。
2. Gaussian noise adds noise calibrated to C to the summed gradients.
   Gaussian noise 依 C 校準後，加入 summed gradients。

An \((\varepsilon,\delta)\)-DP bound limits how much the output distribution can change when one example is added or removed. Lower \(\varepsilon\) generally requires more noise, so the report must show privacy budget and utility together.

\((\varepsilon,\delta)\)-DP bound 會限制加入或移除一筆樣本時，輸出分布最多能改變多少。較低的 \(\varepsilon\) 通常需要更多 noise，因此報告必須同時呈現 privacy budget 與 utility。

4. Secure aggregation hides individual updates｜Secure aggregation 隱藏個別更新

DP limits what the final model can reveal. Secure aggregation addresses a different threat: an aggregation server inspecting each client’s update. Clients mask their updates so the server can recover only the sum; the individual masks cancel when aggregated.⁶

DP 限制 final model 能洩漏的資訊。Secure aggregation 處理另一個威脅：aggregation server 檢查每個 client 的 update。Clients 先遮罩更新，server 只能還原總和；個別 masks 會在聚合時互相抵銷。⁶

The two mechanisms are complementary. DP bounds model leakage, while secure aggregation limits what the server observes during training.

兩種機制互補：DP 約束 model leakage，secure aggregation 則限制 server 在訓練期間能看到的內容。

5. Implementation checks｜實作檢查

A clipping bug or miscalibrated noise multiplier can invalidate the intended privacy bound. The lab therefore checks four things:

Clipping bug 或校準錯誤的 noise multiplier 都可能使預期的 privacy bound 失效，因此 lab 會檢查四件事：

• Implementations cover FedAvg, FedProx, SCAFFOLD, FedPer, Byzantine-robust aggregation, FedAdam, and FedLoRA.
  實作涵蓋 FedAvg、FedProx、SCAFFOLD、FedPer、Byzantine-robust aggregation、FedAdam 與 FedLoRA。
• DP tests check clipping and noise behavior rather than only final accuracy.
  DP 測試會檢查 clipping 與 noise 行為，不只看最終 accuracy。
• Literature cross-checks verify that measured behavior is consistent with the cited methods.
  文獻交叉檢查會確認實測行為是否和引用方法一致。
• The 33/33 tests and negative results remain in the repository with reproduction commands.
  33／33 項測試、負面結果與重現命令都保留在 repository。

→ Implementations, tests, and negative results:
實作、測試與負面結果： github.com/waynehacking8/federated-learning-lab

---

1. Federated Learning Lab, the primary artifact for implementations, tests, and measured negative results discussed here.／本文實作、測試與負面結果的主要來源。 ↩↩

2. McMahan et al., “Communication-Efficient Learning of Deep Networks from Decentralized Data”.／FedAvg 原始論文。 ↩↩

3. Li et al., “Federated Optimization in Heterogeneous Networks”.／FedProx 論文。 ↩↩

4. Karimireddy et al., “SCAFFOLD: Stochastic Controlled Averaging for Federated Learning”.／SCAFFOLD 論文。 ↩↩

5. Abadi et al., “Deep Learning with Differential Privacy”.／DP-SGD 論文。 ↩↩

6. Bonawitz et al., “Practical Secure Aggregation for Privacy-Preserving Machine Learning”.／Secure aggregation 論文。 ↩↩