Understanding the Connection Management System
Connection Pool & Service Principal Management
The Dynamics V2 library implements a dedicated connection pool per service principal to maximize throughput while maintaining strict isolation between service principal credentials. This approach is particularly important for proper OAuth token management and query logging attribution.
┌──────────────────────────── Service Principal Layer ─────────────────────────┐
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ SP1 │ │ SP2 │ │ SPN │ │
│ │ Client ID 1 │ │ Client ID 2 │ │ Client ID N │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
└─────────┼────────────────────┼────────────────────┼──────────────────────────┘
│ │ │
┌─────────┼────────────────────┼────────────────────┼──────────────────────────┐
│ │ │ │ Connection │
│ ┌─────┴──────┐ ┌─────┴──────┐ ┌─────┴──────┐ Factory │
│ │ Pool for │ │ Pool for │ │ Pool for │ │
│ │ Client ID 1│ │ Client ID 2│ │ Client ID N│ (Round-Robin │
│ └─────┬──────┘ └─────┬──────┘ └─────┬──────┘ Selection) │
│ │ │ │ │
└─────────┼────────────────────┼────────────────────┼──────────────────────────┘
│ │ │
┌─────────┼────────────────────┼────────────────────┼──────────────────────────┐
│ │ │ │ Connection │
│ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐ Management │
│ │ Conn 1 │ │ Conn 2 │ │ Conn N │ │
│ │memory://│ │memory://│ │memory://│ (OAuth Token per │
│ │ SP1 │ │ SP2 │ │ SPN │ Service Principal) │
│ └────┬────┘ └────┬────┘ └────┬────┘ │
└─────────┼────────────────────┼────────────────────┼──────────────────────────┘
│ │ │
└──────────┬─────────┴───────────┬────────┘
│ │
┌────────────────────┼─────────────────────┼───────────────────────────────────┐
│ ┌───────┴─────────────────────┴───────┐ get_connection() │
│ │ Connection Manager │ │
│ └─────────────────────────────────────┘ (Context Management) │
└──────────────────────────────────────────────────────────────────────────────┘
Connection Distribution Flow
The Dynamics V2 library uses a sophisticated multi-layered system to distribute database operations across service principals and execute them efficiently in parallel. Here's how it works:
┌─────────────────┐
│ Incoming Data │
│ (DataFrame) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Split into │
│ Operations │
└────────┬────────┘
│
▼
┌──────────────────────────────────┐
│ ThreadPoolExecutor │
│ │
│ ┌─────────┐ ┌─────────┐ │
│ │Worker 1 │ │Worker 2 │ ... │
│ └────┬────┘ └────┬────┘ │
│ │ │ │
└─────────┼────────────┼───────────┘
│ │
▼ ▼
┌──────────────────────────────────┐
│ Round-Robin SP Selection │
│ │
│ SP1 → SP2 → SP3 → ... → SPN → SP1│
└─────────┬────────────┬───────────┘
│ │
▼ ▼
┌──────────────────────────────────┐
│ Connection Pools │
│ ┌─────┐ ┌─────┐ ┌─────┐ │
│ │SP1 │ │SP2 │... │SPN │ │
│ │Pool │ │Pool │ │Pool │ │
│ └──┬──┘ └──┬──┘ └──┬──┘ │
└─────┼────────┼──────────┼────────┘
│ │ │
▼ ▼ ▼
┌──────────────────────────────────┐
│ Dynamics CRM Client │
└──────────────────────────────────┘
When your data enters the system, here's what happens:
-
Data Batching
Your DataFrame is first split into manageable operations (inserts, updates, etc.). -
Thread Pool Allocation
These operations are then handed to a thread pool executor. Each worker in the pool then handles identifying the relevant insert/update/stored-procedure calls required for the operation, retrieving a connection from the pool, making the api calls to the Dynamics CRM SOAP API, and handling any retry on rate limit failures (if configured).
Operation Queue Thread Pool Connections
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Op 1 │ → │ Worker 1 │ → │ SP1 Pool │
│ Op 2 │ → │ Worker 2 │ → │ SP2 Pool │
│ Op 3 │ → │ Worker 3 │ → │ SP3 Pool │
│ ... │ │ ... │ │ ... │
└──────────┘ └──────────┘ └──────────┘
- Service Principal Distribution
As each worker picks up an operation, it needs a connection to Dynamics. This is where the round-robin system comes in:
Time → Worker1 Worker2 Worker3 Worker4
t1 SP1 SP2 SP3 SP4
t2 SP5 SP1 SP2 SP3
t3 SP4 SP5 SP1 SP2
- Connection Management
Each service principal has its own dedicated pool of connections. When a worker needs to execute an operation:
Worker Request Flow:
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Worker │ │ SP │ │Connection│
│ Thread │ → │Selection │ → │ Pool │
└──────────┘ └──────────┘ └──────────┘
↑ │
└─────────────────────────────────┘
Return connection when done
This creates a continuous flow where:
- Multiple workers execute operations in parallel
- Each worker gets a fair share of service principal connections
- No service principal gets overloaded
- Connections are efficiently reused
The system automatically balances the load across all available service principals while maintaining orderly execution.
If configured, when a rate limit is hit on one service principal, that SP's connections will naturally pause while others continue working.
Rate Limit Scenario:
SP1: [RATE LIMITED] → Operations redirect to other SPs
SP2: [PROCESSING] → Continues normal operation
SP3: [PROCESSING] → Continues normal operation
SP4: [PROCESSING] → Continues normal operation
This resilient design means your operations continue processing even when individual service principals hit temporary limits, maximizing overall throughput while respecting Dynamics CRM's rate limiting requirements.
Updated about 17 hours ago