4.1.4. Edge Deployment with SageMaker Neo
💡 First Principle: Edge deployment runs models on devices with limited compute (IoT sensors, cameras, mobile phones) instead of in the cloud. This eliminates network latency and enables predictions where internet connectivity is unreliable. SageMaker Neo compiles models to run efficiently on specific edge hardware—and the exam tests when edge deployment is the right choice.
Neo takes a trained model and compiles it for a target hardware platform (ARM, Intel, NVIDIA, Qualcomm). The compiled model runs faster and uses less memory than the original, enabling deployment on resource-constrained devices. The compilation process optimizes the model's computation graph for the target hardware's instruction set, often achieving 2-10x performance improvements without changing the model architecture.
The edge deployment workflow on AWS:
- Train the model in SageMaker (cloud)
- Compile with Neo for the target hardware platform
- Package the compiled model with Neo runtime (DLR — Deep Learning Runtime)
- Deploy to the edge device using AWS IoT Greengrass (for fleet management) or direct deployment
When edge deployment makes sense: Low-latency requirements at the device level (autonomous vehicles need sub-10ms predictions), intermittent or no connectivity (remote oil rigs, agricultural sensors), privacy constraints (medical imaging data that can't leave the hospital), or high-bandwidth data where streaming raw data to cloud is impractical (security cameras generating 30 FPS video at dozens of locations).
AWS IoT Greengrass manages edge ML at scale. It deploys models to fleets of devices, manages model versioning, and provides local Lambda execution. When the exam describes deploying a model to "thousands of devices," Greengrass is the management layer and Neo provides the compiled inference engine.
⚠️ Exam Trap: Edge deployment via Neo is for inference only—training always happens in the cloud. If a question mentions "training on edge devices," that's not a valid Neo use case. Also, Neo has framework and hardware limitations—not every model can be compiled for every target. Currently supported frameworks include TensorFlow, PyTorch, MXNet, and XGBoost, but custom frameworks or unusual architectures may require manual optimization.
Reflection Question: A manufacturing company needs to detect defects in products on the assembly line using cameras. Each camera needs to process 30 frames per second. Internet connectivity is unreliable. Should inference run in the cloud or on the edge?
