I am trying to get throughput control to work with Cosmos DB Spark connector, but I don’t see the new containers created.
Here is the code:
val changeFeedCfg = Map("spark.cosmos.accountEndpoint" -> cosmosEndpoint,"spark.cosmos.accountKey" -> cosmosMasterKey,"spark.cosmos.database" -> cosmosDatabaseName,"spark.cosmos.container" -> cosmosContainerName,"spark.cosmos.read.inferSchema.enabled" -> "false","spark.cosmos.changeFeed.startFrom" -> "Beginning","spark.cosmos.changeFeed.mode" -> "Incremental","spark.cosmos.changeFeed.itemCountPerTriggerHint" -> "100000"//optional configuration for throughput control// "spark.cosmos.throughputControl.enabled" -> "true",// "spark.cosmos.throughputControl.name" -> "SourceContainerThroughputControl",// "spark.cosmos.throughputControl.targetThroughputThreshold" -> "0.30",// "spark.cosmos.throughputControl.globalControl.database" -> "database-v4",// "spark.cosmos.throughputControl.globalControl.container" -> "ThroughputControl"
"spark.cosmos.throughputControl.useGatewayMode" -> "true")`
I had tried the code and it is not working as excepted.
>Solution :
I see you are using Gateway mode.
"spark.cosmos.throughputControl.useGatewayMode" -> "true"
Currently we do not have support for throughput control in gateway mode
https://github.com/Azure/azure-sdk-for-java/issues/34229