Understanding Rate Limit Handling
Two-Layer Retry System
The Dynamics V2 library implements a comprehensive system to handle Dynamics CRM's rate limiting. Rather than failing operations when rate limits are hit, the system intelligently retries operations while respecting CRM's constraints. This ensures reliable data processing even under heavy load.
Key aspects of the retry mechanism:
-
Multi-Layer Protection
- Immediate retry for transient rate limits
- Deferred retry for persistent issues
- Configurable parameters at both layers
-
Smart Waiting
- Respects Dynamics' retry-after headers
- Uses exponential backoff with jitter
- Supports batch-level delays
-
Resource Management
- Clones batch managers for deferred retries
- Maintains operation dependencies
- Tracks retry attempts and deadlines
-
Metrics and Logging
- Tracks successful/failed retries
- Records retry attempts and delays
- Provides detailed error information
Rate Limits in Dynamics CRM
Dynamics CRM enforces several types of rate limits that we need to handle:
- Concurrent request limits (max 52 concurrent connections)
- Execution time limits (per time window)
- Request count limits (max requests per time window)
When these limits are hit, CRM returns specific rate limit errors with retry-after guidance. Our retry system is built around handling these gracefully.
The package implements two layers of retry:
- Immediate Retry: Using tenacity decorator for quick retries of rate-limited operations
- Deferred Retry: Queuing failed operations for later retry with longer delays
To handle rate limits effectively, we implement two distinct retry layers:
Immediate Retry
Operation ──────┐ ┌────────────────────────────────────────────────┐
Attempt │ │ │
▼ ▼ │
┌──────────────┐ ┌───────────────┐ ┌────────────┐ │
│ Rate Limit │ Yes │ Check Retry │ Yes │ Queue │ │
│ Error? │────▶│ Available? │────▶│ For Retry │ │
└──────────────┘ └───────────────┘ └────────────┘ │
│ │ │ │
No │ No │ │ │
▼ ▼ ▼ │
┌──────────────┐ ┌───────────────┐ ┌────────────┐ │
│ Other │ Yes │ Log Error │ │ Wait Retry │ │
│ Error? │────▶│ & Quarantine │ │ Delay │ │
└──────────────┘ └───────────────┘ └────────────┘ │
No │ │ │
▼ ▼ │
┌──────────────┐ ┌────────────┐ │
│ Success │ │ Retry │────┘
└──────────────┘ └────────────┘
When an operation first hits a rate limit:
- The system attempts quick retries with short delays
- Respects Dynamics CRM's retry-after headers
- Uses exponential backoff with jitter to prevent operation bunching
- Makes up to three attempts before escalating to deferred retry
Immediate Retry Configuration
# Configuration settings
retry_config = {
"advanced.retry.enabled": True, # Enable automatic retry
"advanced.retry.max_attempts": 3, # Maximum retry attempts
"advanced.retry.base_delay": 2, # Base delay in seconds
"advanced.retry.max_delay": 60, # Maximum delay in seconds
"advanced.retry.jitter": 1.5 # Jitter factor
}
Deferred Retry
For operations that couldn't be resolved through immediate retry:
┌─────────────────────────────────────── WHILE LOOP ────────────────────────────────────────┐
│ │
│ ┌─────────────────┐ │
│ ┌───────────────▶│ Start │◀──────────────────────┐ │
│ │ └─────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────┐ Yes │ │
│ │ │ Queue Empty? │──────────┐ │ │
│ │ └─────────────────┘ │ │ │
│ │ │ ▼ │ │
│ │ No │ ┌────────────┐ │ │
│ │ │ │ Exit │ │ │
│ │ ▼ │ Loop │ │ │
│ │ ┌─────────────────┐ └────────────┘ │ │
│ │ │ Get Operations │ │ │
│ │ │ Ready for Retry │ │ │
│ │ └─────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────┐ │ │
│ │ │ Any Ready? │ │ │
│ │ └─────────────────┘ │ │
│ │ │ │ │ │
│ │ Yes │ No │ │ │
│ │ │ ▼ │ │
│ │ │ ┌─────────────────┐ │ │
│ │ │ │ Get All Retry- │ │ │
│ │ │ │ After Values │ │ │
│ │ │ └─────────────────┘ │ │
│ │ │ │ │ │
│ │ │ ▼ │ │
│ │ │ ┌─────────────────┐ │ │
│ │ │ │ Calculate Mean │ │ │
│ │ │ │ Delay Time │ │ │
│ │ │ └─────────────────┘ │ │
│ │ │ │ │ │
│ │ │ ▼ │ │
│ │ │ ┌─────────────────┐ │ │
│ │ │ │ Sleep for │────────────────┘ │
│ │ │ │ Mean Delay │ │
│ │ │ └─────────────────┘ │
│ │ │ │
│ │ │ │
│ │ ┌─────────────────────┐ │
│ │ │ Max Retries │ Yes │
│ │ │ Exceeded? │────────────────────────────────┐ │
│ │ └─────────────────────┘ │ │
│ │ │ │ │
│ │ No │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────┐ │ │
│ │ │ Process Deferred │ │ │
│ │ │ Operation │ │ │
│ │ └─────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────┐ No ┌───────────────┐ │ │
│ │ │ Error? │───────▶│ Success │ │ │
│ │ └─────────────────────┘ └───────────────┘ │ │
│ │ │ │ │ │
│ │ Yes │ │ │ │
│ │ ▼ ▼ │ │
│ │ ┌─────────────────────┐ ┌───────────────┐ │ │
│ │ │ Rate Limit? │ │ Remove from │ │ │
│ │ └─────────────────────┘ │ Queue │ │ │
│ │ │ │ └───────────────┘ │ │
│ │ Yes │ No │ │ │
│ │ ▼ │ │ │
│ │ ┌──────────────┐ │ │ │
│ │ │ Increment │ │ │ │
│ └──────────│ Attempt & │ │ │ │
│ │ Update Delay │ │ │ │
│ └──────────────┘ │ │ │
│ │ │ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────┐ │ │
│ │ Log & │◀──────────────────────┘ │
│ │ Quarantine │ │
│ └─────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Remove from │ │
│ │ Queue │ │
│ └─────────────────────┘ │
│ │
└───────────────────────────────────────────────────────────────────────────────────────────┘
The deferred system:
- Groups related operations to maintain data consistency
- Implements a 5-minute initial delay
- Processes operations in smaller batches
- Preserves operation order and dependencies
- Automatically quarantines after three failed attempts
Deferred Retry Configuration
deferred_retry_config = {
"advanced.performance.deferred_retry_enabled": True, # Enable deferred retry
"advanced.performance.deferred_batch_size": 1000, # Records per retry batch
"advanced.performance.max_deferred_retries": 3, # Maximum retry attempts
"advanced.performance.deferred_retry_delay": 300, # Initial retry delay (seconds)
}
This two-layered approach balances quick recovery for transient rate limits with proper handling of persistent limiting situations. The system logs retry attempts and final outcomes to the batch metrics table, allowing you to monitor rate limit impacts through your standard logging queries.
When implementing your processing logic, you can configure both layers through the retry/performance settings documented in the configuration guide, allowing you to tune the behavior for your specific needs while maintaining safe interaction with Dynamics CRM.
Updated about 17 hours ago