IBM Cloud/WML/notebooks/model_risk_management/OpenScale and OpenPages model risk management on IBM Cloud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook should be run in a Watson Studio project, using **Default Spark 3.3.x & Python 3.10 or Python 3.9 ** runtime environment. **If you are viewing this in Watson Studio and do not see the required runtime env in the upper right corner of your screen, please update the runtime now.**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "To run this notebook, you must provide the following information.\n",
    "\n",
    "- IBMid and IBM Cloud instance\n",
    "- two (2) instances of IBM Watson Machine Learning\n",
    "- an instance of IBM Watson OpenScale\n",
    "- authentication credentials for IBM OpenPages\n",
    "- a model in IBM OpenPages for which you have copied the `model_id` value from the model URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Provision services and configure credentials\n",
    "\n",
    "If you have not already, provision an instance of IBM Watson OpenScale and two instances of IBM Watson Machine Learning using the Cloud catalog.\n",
    "\n",
    "The credentials of the OpenPages system shall be provided to the Beta participants.\n",
    "\n",
    "Your Cloud API key can be generated by going to the Users section of the Cloud console. From that page, click your name, scroll down to the API Keys section, and click Create an IBM Cloud API key. Give your key a name and click Create, then copy the created key and paste it below.\n",
    "\n",
    "NOTE: You can also get OpenScale API_KEY using IBM CLOUD CLI.\n",
    "\n",
    "How to install IBM Cloud (bluemix) console: [Instructions](https://console.bluemix.net/docs/cli/reference/ibmcloud/download_cli.html#install_use)\n",
    "\n",
    " \n",
    "**Connection to WML**\n",
    "    \n",
    "Authenticate the Watson Machine Learning service on IBM Cloud. You need to provide platform api_key and instance location.\n",
    "\n",
    "You can use IBM Cloud CLI to retrieve platform API Key and instance location.\n",
    "\n",
    "API Key can be generated in the following way:\n",
    "```\n",
    "ibmcloud login\n",
    "ibmcloud iam api-key-create API_KEY_NAME\n",
    "In result, get the value of api_key from the output.\n",
    "```\n",
    "Location of your WML instance can be retrieved in the following way:\n",
    "```\n",
    "ibmcloud login --apikey API_KEY -a https://cloud.ibm.com\n",
    "ibmcloud resource service-instances\n",
    "ibmcloud resource service-instance WML_INSTANCE_NAME\n",
    "ibmcloud resource service-instance COS_INSTANCE_NAME\n",
    "```\n",
    "In result, get the value of location from the output.\n",
    "\n",
    "In the output, you can also get:\n",
    "\n",
    "- **name of the service instance\n",
    "CRN (ID) and Name (name)**\n",
    "\n",
    "that can be used in next steps.\n",
    "\n",
    "Tip: Your Cloud API key can be generated by going to the Users section of the Cloud console. From that page, click your name, scroll down to the API Keys section, and click Create an IBM Cloud API key. Give your key a name and click Create, then copy the created key and paste it below.\n",
    "\n",
    "You can also get service specific apikey by going to the Service IDs section of the Cloud Console. From that page, click Create, then copy the created key and paste it below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#############################################################################\n",
    "# Paste your IBM Cloud API key and WML NAME,CRN in the following field and then run this cell.\n",
    "#############################################################################\n",
    "CLOUD_API_KEY = \"***\"\n",
    "WML_INSTANCE_NAME=\"***\"\n",
    "WML_CRN=\"***\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "COS_API_KEY_ID = \"***\"\n",
    "COS_RESOURCE_CRN = \"***\" # eg \"crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003abfb5d29761c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::\"\n",
    "COS_ENDPOINT = \"***\" # Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET_NAME = \"***\" #example: \"credit-risk-training-data\" \n",
    "training_data_file_name=\"german_credit_data_biased_training.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "WML_CREDENTIALS = {\n",
    "                   \"url\": \"https://us-south.ml.cloud.ibm.com\",\n",
    "                   \"apikey\": CLOUD_API_KEY\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DB_CREDENTIALS=None\n",
    "#DB_CREDENTIALS= {\"hostname\":\"\",\"username\":\"\",\"password\":\"\",\"database\":\"\",\"port\":\"\",\"ssl\":True,\"sslmode\":\"\",\"certificate_base64\":\"\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "KEEP_MY_INTERNAL_POSTGRES = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IAM_URL=\"https://iam.ng.bluemix.net/oidc/token\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integration system credentials (IBM OpenPages MRG)\n",
    "As part of the closed beta, you were sent an email message from IBM that provides the URL, username, and password of an IBM OpenPages system that you can use. Replace the following credentials with the ones that you received in that email message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################################################################################\n",
    "# Paste your IBM OpenPages credentials in the following fields and then run this cell.\n",
    "######################################################################################\n",
    "OPENPAGES_CREDENTIALS = {\n",
    "    \"url\":\"***\",\n",
    "    \"username\": \"***\",\n",
    "    \"password\": \"***\"\n",
    "  }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model ID\n",
    "For the model that you create in IBM OpenPages, copy the `model_id` from the model URL in the browser address bar. For example, if you create a model and view it in the OpenPages UI, you might see a URL such as \"openpages_url/app/jspview/react/grc/task-view/7916?\". In this example, the `model_id` is **7916**. Paste that value into the following code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "####################################################################\n",
    "# Paste your model ID in the following field and then run this cell.\n",
    "####################################################################\n",
    "openpages_model_id=\"***\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package installation\n",
    "The following opensource packages must be installed into this notebook instance so that they are available to use during processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade pyspark==3.2.0 --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you are not using IBM Watson Studio then please install the following dependencies\n",
    "\n",
    "#!pip install --upgrade pandas==1.2.3 --no-cache | tail -n 1\n",
    "#!pip install --upgrade requests==2.23 --no-cache | tail -n 1\n",
    "#!pip install --upgrade numpy==1.20.3 --user --no-cache | tail -n 1\n",
    "#!pip install SciPy --no-cache | tail -n 1\n",
    "#!pip install lime --no-cache | tail -n 1\n",
    "#!pip install pyspark==3.3.0 --no-cache | tail -n 1\n",
    "\n",
    "!pip install --upgrade ibm-watson-machine-learning --user | tail -n 1\n",
    "!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "import base64\n",
    "from requests.auth import HTTPBasicAuth\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the training data from Github\n",
    "So you don't have to manually generate training data, we've provided a sample and placed it in a publicly available Github repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm german_credit_data_biased_training.csv\n",
    "!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/german_credit_data_biased_training.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ibm_boto3\n",
    "from ibm_botocore.client import Config, ClientError\n",
    "\n",
    "cos_client = ibm_boto3.resource(\"s3\",\n",
    "    ibm_api_key_id=COS_API_KEY_ID,\n",
    "    ibm_service_instance_id=COS_RESOURCE_CRN,\n",
    "    ibm_auth_endpoint=\"https://iam.bluemix.net/oidc/token\",\n",
    "    config=Config(signature_version=\"oauth\"),\n",
    "    endpoint_url=COS_ENDPOINT\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(training_data_file_name, \"rb\") as file_data:\n",
    "    cos_client.Object(BUCKET_NAME, training_data_file_name).upload_fileobj(\n",
    "        Fileobj=file_data\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the Spark Credit Risk Model to Watson Machine Learning\n",
    "\n",
    "The following cell deploys the Spark version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. You'll notice that this version of the German Credit Risk model has an auc-roc score around 71%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_machine_learning import APIClient\n",
    "\n",
    "wml_client = APIClient(WML_CREDENTIALS)\n",
    "wml_client.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wml_client.spaces.list(limit=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "space_name =\"pre-prod-space\"\n",
    "spaces = wml_client.spaces.get_details()['resources']\n",
    "preprod_space_id = None\n",
    "for space in spaces:\n",
    "    if space['entity']['name'] == space_name:\n",
    "        preprod_space_id = space[\"metadata\"][\"id\"]\n",
    "if preprod_space_id is None:\n",
    "    preprod_space_id = wml_client.spaces.store(\n",
    "        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name,\n",
    "                   wml_client.spaces.ConfigurationMetaNames.STORAGE: {\"resource_crn\":COS_RESOURCE_CRN},\n",
    "                   wml_client.spaces.ConfigurationMetaNames.COMPUTE: {\"name\": WML_INSTANCE_NAME,\n",
    "                                            \"crn\": WML_CRN}})[\"metadata\"][\"id\"]\n",
    "wml_client.set.default_space(preprod_space_id)\n",
    "print(preprod_space_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deploy_credit_risk_spark_model(wml_credentials, model_name, deployment_name,space_id):\n",
    "\n",
    "    import numpy \n",
    "    numpy.version.version\n",
    "\n",
    "    import pandas as pd\n",
    "    import json\n",
    "\n",
    "    from pyspark import SparkContext, SQLContext\n",
    "    from pyspark.ml import Pipeline\n",
    "    from pyspark.ml.classification import RandomForestClassifier,GBTClassifier\n",
    "    from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "    from pyspark.ml.feature import StringIndexer, VectorAssembler, IndexToString\n",
    "    from pyspark.sql.types import StructType, DoubleType, StringType, ArrayType\n",
    "\n",
    "    from pyspark.sql import SparkSession\n",
    "    from pyspark import SparkFiles\n",
    "\n",
    "    spark = SparkSession.builder.getOrCreate()\n",
    "    pd_data = pd.read_csv(\"german_credit_data_biased_training.csv\", sep=\",\", header=0)\n",
    "    spark_df = spark.read.csv(path=\"german_credit_data_biased_training.csv\", sep=\",\", header=True, inferSchema=True)\n",
    "    spark_df.head()\n",
    "\n",
    "    (train_data, test_data) = spark_df.randomSplit([0.9, 0.1], 24)\n",
    "    print(\"Number of records for training: \" + str(train_data.count()))\n",
    "    print(\"Number of records for evaluation: \" + str(test_data.count()))\n",
    "\n",
    "    si_CheckingStatus = StringIndexer(inputCol='CheckingStatus', outputCol='CheckingStatus_IX')\n",
    "    si_CreditHistory = StringIndexer(inputCol='CreditHistory', outputCol='CreditHistory_IX')\n",
    "    si_LoanPurpose = StringIndexer(inputCol='LoanPurpose', outputCol='LoanPurpose_IX')\n",
    "    si_ExistingSavings = StringIndexer(inputCol='ExistingSavings', outputCol='ExistingSavings_IX')\n",
    "    si_EmploymentDuration = StringIndexer(inputCol='EmploymentDuration', outputCol='EmploymentDuration_IX')\n",
    "    si_Sex = StringIndexer(inputCol='Sex', outputCol='Sex_IX')\n",
    "    si_OthersOnLoan = StringIndexer(inputCol='OthersOnLoan', outputCol='OthersOnLoan_IX')\n",
    "    si_OwnsProperty = StringIndexer(inputCol='OwnsProperty', outputCol='OwnsProperty_IX')\n",
    "    si_InstallmentPlans = StringIndexer(inputCol='InstallmentPlans', outputCol='InstallmentPlans_IX')\n",
    "    si_Housing = StringIndexer(inputCol='Housing', outputCol='Housing_IX')\n",
    "    si_Job = StringIndexer(inputCol='Job', outputCol='Job_IX')\n",
    "    si_Telephone = StringIndexer(inputCol='Telephone', outputCol='Telephone_IX')\n",
    "    si_ForeignWorker = StringIndexer(inputCol='ForeignWorker', outputCol='ForeignWorker_IX')\n",
    "    si_Label = StringIndexer(inputCol=\"Risk\", outputCol=\"label\").fit(spark_df)\n",
    "    label_converter = IndexToString(inputCol=\"prediction\", outputCol=\"predictedLabel\", labels=si_Label.labels)\n",
    "\n",
    "    va_features = VectorAssembler(\n",
    "    inputCols=[\"CheckingStatus_IX\", \"CreditHistory_IX\", \"LoanPurpose_IX\", \"ExistingSavings_IX\",\n",
    "               \"EmploymentDuration_IX\", \"Sex_IX\", \"OthersOnLoan_IX\", \"OwnsProperty_IX\", \"InstallmentPlans_IX\",\n",
    "               \"Housing_IX\", \"Job_IX\", \"Telephone_IX\", \"ForeignWorker_IX\", \"LoanDuration\", \"LoanAmount\",\n",
    "               \"InstallmentPercent\", \"CurrentResidenceDuration\", \"LoanDuration\", \"Age\", \"ExistingCreditsCount\",\n",
    "               \"Dependents\"], outputCol=\"features\")\n",
    "\n",
    "    classifier=GBTClassifier(featuresCol=\"features\")\n",
    "\n",
    "    pipeline = Pipeline(\n",
    "    stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker,\n",
    "            si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,\n",
    "            si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])\n",
    "\n",
    "    model = pipeline.fit(train_data)\n",
    "    predictions = model.transform(test_data)\n",
    "    evaluator = BinaryClassificationEvaluator(rawPredictionCol=\"prediction\")\n",
    "    auc = evaluator.evaluate(predictions)\n",
    "\n",
    "    print(\"Accuracy = %g\" % auc)\n",
    "\n",
    "    from ibm_watson_machine_learning import APIClient\n",
    "\n",
    "    wml_client = APIClient(WML_CREDENTIALS)\n",
    "    wml_client.version\n",
    "    wml_client.set.default_space(space_id)\n",
    "    \n",
    "\n",
    "    # Remove existing model and deployment\n",
    "    MODEL_NAME=model_name\n",
    "    DEPLOYMENT_NAME=deployment_name\n",
    "\n",
    "    deployments_list = wml_client.deployments.get_details()\n",
    "    for deployment in deployments_list[\"resources\"]:\n",
    "        model_id = deployment[\"entity\"][\"asset\"][\"id\"]\n",
    "        deployment_id = deployment[\"metadata\"][\"id\"]\n",
    "        if deployment[\"metadata\"][\"name\"] == DEPLOYMENT_NAME:\n",
    "            print(\"Deleting deployment id\", deployment_id)\n",
    "            wml_client.deployments.delete(deployment_id)\n",
    "            print(\"Deleting model id\", model_id)\n",
    "            wml_client.repository.delete(model_id)\n",
    "    wml_client.repository.list_models()\n",
    "    \n",
    "    datasource_type = wml_client.connections.get_datasource_type_uid_by_name('bluemixcloudobjectstorage')\n",
    "    conn_meta_props= {\n",
    "        wml_client.connections.ConfigurationMetaNames.NAME: \"Connection My COS \",\n",
    "        wml_client.connections.ConfigurationMetaNames.DATASOURCE_TYPE: datasource_type,\n",
    "        wml_client.connections.ConfigurationMetaNames.DESCRIPTION: \"Connection to my COS\",\n",
    "        wml_client.connections.ConfigurationMetaNames.PROPERTIES: {\n",
    "            'bucket': BUCKET_NAME,\n",
    "            'api_key': COS_API_KEY_ID,\n",
    "            'resource_instance_id': COS_RESOURCE_CRN,\n",
    "            'iam_url': \"https://iam.ng.bluemix.net/oidc/token\",\n",
    "            'url': COS_ENDPOINT\n",
    "        }\n",
    "    }\n",
    "\n",
    "    conn_details = wml_client.connections.create(meta_props=conn_meta_props)\n",
    "    connection_id = wml_client.connections.get_uid(conn_details)\n",
    "\n",
    "    training_data_references = [\n",
    "        {\n",
    "            \"id\": \"German Credit Risk\", \n",
    "            \"type\": \"connection_asset\",\n",
    "            \"connection\": {\n",
    "                \"id\": connection_id,\n",
    "                \"href\": \"/v2/connections/\" + connection_id + \"?space_id=\" + space_id\n",
    "\n",
    "            },\n",
    "            \"location\": {\n",
    "                \"bucket\": BUCKET_NAME,\n",
    "                \"file_name\": training_data_file_name\n",
    "            }\n",
    "        }    \n",
    "    ]\n",
    "\n",
    "    # Save Model\n",
    "    software_spec_uid = wml_client.software_specifications.get_id_by_name(\"spark-mllib_3.3\")\n",
    "    print(\"Software Specification ID: {}\".format(software_spec_uid))\n",
    "    model_props = {\n",
    "            wml_client._models.ConfigurationMetaNames.NAME:\"{}\".format(MODEL_NAME),\n",
    "            wml_client._models.ConfigurationMetaNames.TYPE: \"mllib_3.3\",\n",
    "            wml_client._models.ConfigurationMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,\n",
    "            wml_client._models.ConfigurationMetaNames.LABEL_FIELD: \"Risk\",\n",
    "        }\n",
    "\n",
    "    print(\"Storing model ...\")\n",
    "    published_model_details = wml_client.repository.store_model(\n",
    "        model=model, \n",
    "        meta_props=model_props, \n",
    "        training_data=train_data, \n",
    "        pipeline=pipeline)\n",
    "\n",
    "    model_uid = wml_client.repository.get_model_id(published_model_details)\n",
    "    print(\"Done\")\n",
    "    print(\"Model ID: {}\".format(model_uid))\n",
    "\n",
    "\n",
    "    # Deploy model\n",
    "    deployment_details = wml_client.deployments.create(\n",
    "    model_uid, \n",
    "    meta_props={\n",
    "        wml_client.deployments.ConfigurationMetaNames.NAME: \"{}\".format(DEPLOYMENT_NAME),\n",
    "        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "    }\n",
    "    )\n",
    "    scoring_url = wml_client.deployments.get_scoring_href(deployment_details)\n",
    "    deployment_uid=wml_client.deployments.get_id(deployment_details)\n",
    "\n",
    "    print(\"Scoring URL:\" + scoring_url)\n",
    "    print(\"Model id: {}\".format(model_uid))\n",
    "    print(\"Deployment id: {}\".format(deployment_uid))\n",
    "\n",
    "    fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "    values = [\n",
    "      [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "      [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "      [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "      [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "    ]\n",
    "\n",
    "    scoring_payload = {\"input_data\": [{\"fields\": fields, \"values\": values}]}\n",
    "    #print(scoring_payload)\n",
    "\n",
    "    scoring_response = wml_client.deployments.score(deployment_uid, scoring_payload)\n",
    "    print(scoring_response)\n",
    "    \n",
    "    return model_uid, deployment_uid, scoring_url\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the Scikit-Learn Credit Risk Model to Watson Machine Learning\n",
    "\n",
    "The following cell deploys the Scikit-learn version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. This version of the German Credit Risk model has an auc-roc score around 85% and will be called the \"Challenger.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deploy_credit_risk_scikit_model(wml_credentials, model_name, deployment_name,space_id):\n",
    "\n",
    "    import pandas as pd\n",
    "    import json\n",
    "    import sys\n",
    "    import numpy\n",
    "    import sklearn\n",
    "    import sklearn.ensemble\n",
    "    numpy.set_printoptions(threshold=sys.maxsize)\n",
    "    from sklearn.utils.multiclass import type_of_target\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    from sklearn.pipeline import Pipeline\n",
    "    from sklearn.impute import SimpleImputer\n",
    "    from sklearn.preprocessing import StandardScaler, OrdinalEncoder\n",
    "    from sklearn.compose import ColumnTransformer\n",
    "    from sklearn.model_selection import cross_validate\n",
    "    from sklearn.metrics import get_scorer\n",
    "    from sklearn.model_selection import cross_validate\n",
    "    from sklearn.metrics import classification_report\n",
    "\n",
    "    data_df=pd.read_csv (\"german_credit_data_biased_training.csv\")\n",
    "\n",
    "    data_df.head()\n",
    "\n",
    "    target_label_name = \"Risk\"\n",
    "    feature_cols= data_df.drop(columns=[target_label_name])\n",
    "    label= data_df[target_label_name]\n",
    "\n",
    "    # Set model evaluation properties\n",
    "    optimization_metric = 'roc_auc'\n",
    "    random_state = 33\n",
    "    cv_num_folds = 3\n",
    "    holdout_fraction = 0.1\n",
    "\n",
    "    if type_of_target(label.values) in ['multiclass', 'binary']:\n",
    "        X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state, stratify=label.values)\n",
    "    else:\n",
    "        X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state)\n",
    "\n",
    "    # Data preprocessing transformer generation\n",
    "\n",
    "    numeric_transformer = Pipeline(steps=[\n",
    "        ('imputer', SimpleImputer(strategy='median')),\n",
    "        ('scaler', StandardScaler())])\n",
    "    categorical_transformer = Pipeline(steps=[\n",
    "        ('imputer', SimpleImputer(strategy='most_frequent')),\n",
    "        ('OrdinalEncoder', OrdinalEncoder(categories='auto',dtype=numpy.float64 ))])\n",
    "\n",
    "    numeric_features = feature_cols.select_dtypes(include=['int64', 'float64']).columns\n",
    "    categorical_features = feature_cols.select_dtypes(include=['object']).columns\n",
    "\n",
    "    preprocessor = ColumnTransformer(\n",
    "        transformers=[\n",
    "            ('num', numeric_transformer, numeric_features),\n",
    "            ('cat', categorical_transformer, categorical_features)])\n",
    "\n",
    "    # Initiate model and create pipeline\n",
    "    model=sklearn.ensemble.GradientBoostingClassifier()\n",
    "    gbt_pipeline = Pipeline(steps=[('preprocessor', preprocessor), ('classifier', model)])\n",
    "    model_gbt=gbt_pipeline.fit(X_train, y_train)\n",
    "\n",
    "    y_pred = model_gbt.predict(X_holdout)\n",
    "\n",
    "\n",
    "    # Evaluate model performance on test data and Cross validation\n",
    "    scorer = get_scorer(optimization_metric)\n",
    "    scorer(model_gbt,X_holdout, y_holdout)\n",
    "\n",
    "    # Cross validation -3 folds\n",
    "    cv_results = cross_validate(model_gbt,X_train,y_train, scoring={optimization_metric:scorer})\n",
    "    numpy.mean(cv_results['test_' + optimization_metric])\n",
    "\n",
    "    print(classification_report(y_pred, y_holdout))\n",
    "\n",
    "\n",
    "    # Initiate WML\n",
    "    from ibm_watson_machine_learning import APIClient\n",
    "\n",
    "    wml_client = APIClient(WML_CREDENTIALS)\n",
    "    wml_client.version\n",
    "    wml_client.set.default_space(space_id)\n",
    "    \n",
    "\n",
    "    # Remove existing model and deployment\n",
    "    MODEL_NAME=model_name\n",
    "    DEPLOYMENT_NAME=deployment_name\n",
    "\n",
    "    deployments_list = wml_client.deployments.get_details()\n",
    "    for deployment in deployments_list[\"resources\"]:\n",
    "        model_id = deployment[\"entity\"][\"asset\"][\"id\"]\n",
    "        deployment_id = deployment[\"metadata\"][\"id\"]\n",
    "        if deployment[\"metadata\"][\"name\"] == DEPLOYMENT_NAME:\n",
    "            print(\"Deleting deployment id\", deployment_id)\n",
    "            wml_client.deployments.delete(deployment_id)\n",
    "            print(\"Deleting model id\", model_id)\n",
    "            wml_client.repository.delete(model_id)\n",
    "    wml_client.repository.list_models()\n",
    "    \n",
    "    # Store Model\n",
    "    software_spec_uid = wml_client.software_specifications.get_id_by_name(\"runtime-23.1-py3.10\")\n",
    "    print(\"Software Specification ID: {}\".format(software_spec_uid))\n",
    "    \n",
    "    training_data_reference = [\n",
    "                    {\n",
    "                        \"id\": \"credit risk\",\n",
    "                        \"type\": \"s3\",\n",
    "                        \"connection\": {\n",
    "                            \"access_key_id\": COS_API_KEY_ID,\n",
    "                            \"endpoint_url\": COS_ENDPOINT,\n",
    "                            \"resource_instance_id\":COS_RESOURCE_CRN\n",
    "                        },\n",
    "                        \"location\": {\n",
    "                            \"bucket\": BUCKET_NAME,\n",
    "                            \"path\": training_data_file_name,\n",
    "                        }\n",
    "                    }\n",
    "                ]\n",
    "\n",
    "\n",
    "    model_props = {\n",
    "        wml_client._models.ConfigurationMetaNames.NAME:\"{}\".format(MODEL_NAME),\n",
    "        wml_client._models.ConfigurationMetaNames.TYPE: \"scikit-learn_1.1\",\n",
    "        wml_client._models.ConfigurationMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,\n",
    "        wml_client._models.ConfigurationMetaNames.LABEL_FIELD: \"Risk\",\n",
    "    }\n",
    "    \n",
    "    print(\"Storing model ...\")\n",
    "\n",
    "    published_model_details = wml_client.repository.store_model(model=model_gbt, meta_props=model_props, training_data=feature_cols, training_target=label)\n",
    "    model_uid = wml_client.repository.get_model_id(published_model_details)\n",
    "    print(\"Done\")\n",
    "    print(\"Model ID: {}\".format(model_uid))\n",
    "\n",
    "\n",
    "\n",
    "    # Deploy model\n",
    "    print(\"Deploying model...\")\n",
    "    deployment_details = wml_client.deployments.create(\n",
    "        model_uid, \n",
    "        meta_props={\n",
    "            wml_client.deployments.ConfigurationMetaNames.NAME: \"{}\".format(DEPLOYMENT_NAME),\n",
    "            wml_client.deployments.ConfigurationMetaNames.ONLINE: {}\n",
    "        }\n",
    "    )\n",
    "    scoring_url = wml_client.deployments.get_scoring_href(deployment_details)\n",
    "    deployment_uid=wml_client.deployments.get_id(deployment_details)\n",
    "\n",
    "    print(\"Scoring URL:\" + scoring_url)\n",
    "    print(\"Model id: {}\".format(model_uid))\n",
    "    print(\"Deployment id: {}\".format(deployment_uid))\n",
    "\n",
    "    # Sample scoring\n",
    "    fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "    values = [\n",
    "      [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "      [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "      [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "      [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "      [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "    ]\n",
    "\n",
    "    payload_scoring = {\"input_data\": [{\"fields\": fields, \"values\": values}]}\n",
    "    #print(payload_scoring)\n",
    "\n",
    "    scoring_response = wml_client.deployments.score(deployment_uid, payload_scoring)\n",
    "    print(scoring_response)\n",
    "\n",
    "    return model_uid, deployment_uid, scoring_url\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy the models\n",
    "\n",
    "The following cells will deploy both the PreProd and Challenger models into the WML instance that is designated as Pre-Production."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PRE_PROD_MODEL_NAME=\"German Credit Risk Model - PreProd\"\n",
    "PRE_PROD_DEPLOYMENT_NAME=\"German Credit Risk Model - PreProd\"\n",
    "\n",
    "PRE_PROD_CHALLENGER_MODEL_NAME=\"German Credit Risk Model - Challenger\"\n",
    "PRE_PROD_CHALLENGER_DEPLOYMENT_NAME=\"German Credit Risk Model - Challenger\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pre_prod_model_uid, pre_prod_deployment_uid, pre_prod_scoring_url = deploy_credit_risk_spark_model(WML_CREDENTIALS, PRE_PROD_MODEL_NAME, PRE_PROD_DEPLOYMENT_NAME,preprod_space_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "challenger_model_uid, challenger_deployment_uid, challenger_scoring_url = deploy_credit_risk_scikit_model(WML_CREDENTIALS, PRE_PROD_CHALLENGER_MODEL_NAME, PRE_PROD_CHALLENGER_DEPLOYMENT_NAME,preprod_space_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configure OpenScale \n",
    "The notebook will now import the necessary libraries and set up a Python OpenScale client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_cloud_sdk_core.authenticators import IAMAuthenticator,BearerTokenAuthenticator\n",
    "\n",
    "from ibm_watson_openscale import *\n",
    "from ibm_watson_openscale.supporting_classes.enums import *\n",
    "from ibm_watson_openscale.supporting_classes import *\n",
    "\n",
    "\n",
    "authenticator = IAMAuthenticator(apikey=CLOUD_API_KEY)\n",
    "#authenticator = BearerTokenAuthenticator(bearer_token=IAM_TOKEN) ## uncomment if using IAM token\n",
    "wos_client = APIClient(authenticator=authenticator)\n",
    "wos_client.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create schema and datamart\n",
    "\n",
    "### Set up datamart\n",
    "Watson OpenScale uses a database to store payload logs and calculated metrics. If database credentials were not supplied above, the notebook will use the free, internal lite database. If database credentials were supplied, the datamart will be created there unless there is an existing datamart and the KEEP_MY_INTERNAL_POSTGRES variable is set to True. If an OpenScale datamart exists in Db2 or PostgreSQL, the existing datamart will be used and no data will be overwritten.\n",
    "\n",
    "Prior instances of the German Credit model will be removed from OpenScale monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_marts = wos_client.data_marts.list().result.data_marts\n",
    "if len(data_marts) == 0:\n",
    "    if DB_CREDENTIALS is not None:\n",
    "        if SCHEMA_NAME is None: \n",
    "            print(\"Please specify the SCHEMA_NAME and rerun the cell\")\n",
    "\n",
    "        print('Setting up external datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\",\n",
    "                database_configuration=DatabaseConfigurationRequest(\n",
    "                  database_type=DatabaseType.POSTGRESQL,\n",
    "                    credentials=PrimaryStorageCredentialsLong(\n",
    "                        hostname=DB_CREDENTIALS['hostname'],\n",
    "                        username=DB_CREDENTIALS['username'],\n",
    "                        password=DB_CREDENTIALS['password'],\n",
    "                        db=DB_CREDENTIALS['database'],\n",
    "                        port=DB_CREDENTIALS['port'],\n",
    "                        ssl=True,\n",
    "                        sslmode=DB_CREDENTIALS['sslmode'],\n",
    "                        certificate_base64=DB_CREDENTIALS['certificate_base64']\n",
    "                    ),\n",
    "                    location=LocationSchemaName(\n",
    "                        schema_name= SCHEMA_NAME\n",
    "                    )\n",
    "                )\n",
    "             ).result\n",
    "    else:\n",
    "        print('Setting up internal datamart')\n",
    "        added_data_mart_result = wos_client.data_marts.add(\n",
    "                background_mode=False,\n",
    "                name=\"WOS Data Mart\",\n",
    "                description=\"Data Mart created by WOS tutorial notebook\", \n",
    "                internal_database = True).result\n",
    "        \n",
    "    data_mart_id = added_data_mart_result.metadata.id\n",
    "    \n",
    "else:\n",
    "    data_mart_id=data_marts[0].metadata.id\n",
    "    print('Using existing datamart {}'.format(data_mart_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bind WML machine learning instance as Pre-Prod\n",
    "\n",
    "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name \"WML Pre-Prod\" already exists, this code will delete that binding a create a new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SERVICE_PROVIDER_NAME = \"Watson Machine Learning pre-prod openpage\"\n",
    "SERVICE_PROVIDER_DESCRIPTION = \"Added by tutorial WOS notebook.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_providers = wos_client.service_providers.list().result.service_providers\n",
    "for service_provider in service_providers:\n",
    "    service_instance_name = service_provider.entity.name\n",
    "    if service_instance_name == SERVICE_PROVIDER_NAME:\n",
    "        service_provider_id = service_provider.metadata.id\n",
    "        wos_client.service_providers.delete(service_provider_id)\n",
    "        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "added_service_provider_result = wos_client.service_providers.add(\n",
    "        name=SERVICE_PROVIDER_NAME,\n",
    "        description=SERVICE_PROVIDER_DESCRIPTION,\n",
    "        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,\n",
    "        deployment_space_id = preprod_space_id, # use pre-prod space ID\n",
    "        operational_space_id = \"pre_production\",\n",
    "        credentials=WMLCredentialsCloud(\n",
    "            apikey=CLOUD_API_KEY,      ## use `apikey=IAM_TOKEN` if using IAM_TOKEN to initiate client\n",
    "            url=WML_CREDENTIALS[\"url\"],\n",
    "            instance_id=None\n",
    "        ),\n",
    "        background_mode=False\n",
    "    ).result\n",
    "service_provider_id = added_service_provider_result.metadata.id\n",
    "service_provider_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id, deployment_space_id = preprod_space_id).result['resources'][1]\n",
    "asset_deployment_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_asset_details_from_deployment=wos_client.service_providers.get_deployment_asset(data_mart_id=data_mart_id,service_provider_id=service_provider_id,deployment_id=pre_prod_deployment_uid,deployment_space_id=preprod_space_id)\n",
    "model_asset_details_from_deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate an IAM token\n",
    "\n",
    "The following is a function that will generate an IAM access token used to interact with the Watson OpenScale APIs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_access_token():\n",
    "    headers={}\n",
    "    headers[\"Content-Type\"] = \"application/x-www-form-urlencoded\"\n",
    "    headers[\"Accept\"] = \"application/json\"\n",
    "    auth = HTTPBasicAuth(\"bx\", \"bx\")\n",
    "    data = {\n",
    "        \"grant_type\": \"urn:ibm:params:oauth:grant-type:apikey\",\n",
    "        \"apikey\": CLOUD_API_KEY\n",
    "    }\n",
    "    response = requests.post(IAM_URL, data=data, headers=headers, auth=auth)\n",
    "    json_data = response.json()\n",
    "    iam_access_token = json_data['access_token']\n",
    "    return iam_access_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create an integration to IBM OpenPages\n",
    "\n",
    "To push metrics from Watson OpenScale to IBM OpenPages, a communication channel must be created between the two services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "WOS_GUID='***'# use openscale instance GUID here\n",
    "\n",
    "INTEGRATED_SYSTEMS_URL = \"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/integrated_systems\".format(WOS_GUID)\n",
    "\n",
    "payload = {\n",
    "    \"name\": \"OpenPages Connection\",\n",
    "    \"type\": \"open_pages\",\n",
    "    \"description\": \"Integration with OpenPages\",\n",
    "    \"credentials\": OPENPAGES_CREDENTIALS\n",
    "}\n",
    "\n",
    "response = requests.post(INTEGRATED_SYSTEMS_URL, json=payload, headers=headers)\n",
    "json_data = response.json()\n",
    "print(json_data)\n",
    "\n",
    "if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n",
    "    integrated_system_id = json_data[\"metadata\"][\"id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subscriptions\n",
    "### Remove existing PreProd and Challenger credit risk subscriptions\n",
    "This code removes previous subscriptions with name `German Credit Risk Model - PreProd` and `German Credit Risk Model - Challenger` to refresh the monitors with the new model and new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subscriptions = wos_client.subscriptions.list().result.subscriptions\n",
    "for subscription in subscriptions:\n",
    "    sub_model_name = subscription.entity.asset.name\n",
    "    if sub_model_name == PRE_PROD_MODEL_NAME or sub_model_name == PRE_PROD_CHALLENGER_MODEL_NAME :\n",
    "        wos_client.subscriptions.delete(subscription.metadata.id)\n",
    "        print('Deleted existing subscription for model', subscription.entity.asset.asset_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointRequest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pre_prod_subscription_details = wos_client.subscriptions.add(\n",
    "        data_mart_id=data_mart_id,\n",
    "        service_provider_id=service_provider_id,\n",
    "        asset=Asset(\n",
    "            asset_id=model_asset_details_from_deployment[\"entity\"][\"asset\"][\"asset_id\"],\n",
    "            name=model_asset_details_from_deployment[\"entity\"][\"asset\"][\"name\"],\n",
    "            url=model_asset_details_from_deployment[\"entity\"][\"asset\"][\"url\"],\n",
    "            asset_type=AssetTypes.MODEL,\n",
    "            input_data_type=InputDataType.STRUCTURED,\n",
    "            problem_type=ProblemType.BINARY_CLASSIFICATION\n",
    "        ),\n",
    "        deployment=AssetDeploymentRequest(\n",
    "            deployment_id=asset_deployment_details['metadata']['guid'],\n",
    "            name=asset_deployment_details['entity']['name'],\n",
    "            deployment_type= DeploymentTypes.ONLINE,\n",
    "            url=asset_deployment_details['metadata']['url'],\n",
    "            scoring_endpoint=ScoringEndpointRequest(url=pre_prod_scoring_url) # score model without shadow deployment\n",
    "        ),\n",
    "        asset_properties=AssetPropertiesRequest(\n",
    "            label_column='Risk',\n",
    "            probability_fields=['probability'],\n",
    "            prediction_field='predictedLabel',\n",
    "            feature_fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n",
    "            categorical_fields = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"],\n",
    "            training_data_reference=TrainingDataReference(type='cos',\n",
    "                                                          location=COSTrainingDataReferenceLocation(bucket = BUCKET_NAME,\n",
    "                                                                                                    file_name = training_data_file_name),\n",
    "                                                          connection=COSTrainingDataReferenceConnection.from_dict({\n",
    "                                                                        \"resource_instance_id\": COS_RESOURCE_CRN,\n",
    "                                                                        \"url\": COS_ENDPOINT,\n",
    "                                                                        \"api_key\": COS_API_KEY_ID,\n",
    "                                                                        \"iam_url\": IAM_URL})),\n",
    "            training_data_schema=SparkStruct.from_dict(model_asset_details_from_deployment[\"entity\"][\"asset_properties\"][\"training_data_schema\"])\n",
    "        )\n",
    "    ).result\n",
    "pre_prod_subscription_id = pre_prod_subscription_details.metadata.id\n",
    "pre_prod_subscription_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subscribe challenger model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "challenger_asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id, deployment_space_id = preprod_space_id).result['resources'][0]\n",
    "challenger_asset_deployment_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "challenger_model_asset_details_from_deployment=wos_client.service_providers.get_deployment_asset(data_mart_id=data_mart_id,service_provider_id=service_provider_id,deployment_id=challenger_deployment_uid,deployment_space_id=preprod_space_id)\n",
    "challenger_model_asset_details_from_deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "challenger_subscription_details = wos_client.subscriptions.add(\n",
    "        data_mart_id=data_mart_id,\n",
    "        service_provider_id=service_provider_id,\n",
    "        asset=Asset(\n",
    "            asset_id=challenger_model_asset_details_from_deployment[\"entity\"][\"asset\"][\"asset_id\"],\n",
    "            name=challenger_model_asset_details_from_deployment[\"entity\"][\"asset\"][\"name\"],\n",
    "            url=challenger_model_asset_details_from_deployment[\"entity\"][\"asset\"][\"url\"],\n",
    "            asset_type=AssetTypes.MODEL,\n",
    "            input_data_type=InputDataType.STRUCTURED,\n",
    "            problem_type=ProblemType.BINARY_CLASSIFICATION\n",
    "        ),\n",
    "        deployment=AssetDeploymentRequest(\n",
    "            deployment_id=challenger_asset_deployment_details['metadata']['guid'],\n",
    "            name=challenger_asset_deployment_details['entity']['name'],\n",
    "            deployment_type= DeploymentTypes.ONLINE,\n",
    "            url=asset_deployment_details['metadata']['url'],\n",
    "            scoring_endpoint=ScoringEndpointRequest(url=challenger_scoring_url) # score model without shadow deployment\n",
    "        ),\n",
    "        asset_properties=AssetPropertiesRequest(\n",
    "            label_column='Risk',\n",
    "            probability_fields=['probability'],\n",
    "            prediction_field='prediction',\n",
    "            feature_fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"],\n",
    "            categorical_fields = [\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"],\n",
    "            training_data_reference=TrainingDataReference(type='cos',\n",
    "                                                          location=COSTrainingDataReferenceLocation(bucket = BUCKET_NAME,\n",
    "                                                                                                    file_name = training_data_file_name),\n",
    "                                                          connection=COSTrainingDataReferenceConnection.from_dict({\n",
    "                                                                        \"resource_instance_id\": COS_RESOURCE_CRN,\n",
    "                                                                        \"url\": COS_ENDPOINT,\n",
    "                                                                        \"api_key\": COS_API_KEY_ID,\n",
    "                                                                        \"iam_url\": IAM_URL})),\n",
    "            training_data_schema=SparkStruct.from_dict(challenger_model_asset_details_from_deployment[\"entity\"][\"asset_properties\"][\"training_data_schema\"])\n",
    "        )\n",
    "    ).result\n",
    "challenger_subscription_id = challenger_subscription_details.metadata.id\n",
    "challenger_subscription_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score the model so we can configure monitors\n",
    "Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "values = [\n",
    "  [\"no_checking\",13,\"credits_paid_to_date\",\"car_new\",1343,\"100_to_500\",\"1_to_4\",2,\"female\",\"none\",3,\"savings_insurance\",46,\"none\",\"own\",2,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",24,\"prior_payments_delayed\",\"furniture\",4567,\"500_to_1000\",\"1_to_4\",4,\"male\",\"none\",4,\"savings_insurance\",36,\"none\",\"free\",2,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",26,\"all_credits_paid_back\",\"car_new\",863,\"less_100\",\"less_1\",2,\"female\",\"co-applicant\",2,\"real_estate\",38,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",14,\"no_credits\",\"car_new\",2368,\"less_100\",\"1_to_4\",3,\"female\",\"none\",3,\"real_estate\",29,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"0_to_200\",4,\"no_credits\",\"car_new\",250,\"less_100\",\"unemployed\",2,\"female\",\"none\",3,\"real_estate\",23,\"none\",\"rent\",1,\"management_self-employed\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",17,\"credits_paid_to_date\",\"car_new\",832,\"100_to_500\",\"1_to_4\",2,\"male\",\"none\",2,\"real_estate\",42,\"none\",\"own\",1,\"skilled\",1,\"none\",\"yes\"],\n",
    "  [\"no_checking\",33,\"outstanding_credit\",\"appliances\",5696,\"unknown\",\"greater_7\",4,\"male\",\"co-applicant\",4,\"unknown\",54,\"none\",\"free\",2,\"skilled\",1,\"yes\",\"yes\"],\n",
    "  [\"0_to_200\",13,\"prior_payments_delayed\",\"retraining\",1375,\"100_to_500\",\"4_to_7\",3,\"male\",\"none\",3,\"real_estate\",37,\"none\",\"own\",2,\"management_self-employed\",1,\"none\",\"yes\"]\n",
    "]\n",
    "\n",
    "payload_scoring = {\"input_data\": [{\"fields\": fields, \"values\": values}]}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(5)\n",
    "preprod_payload_data_set_id = None\n",
    "preprod_payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=pre_prod_subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if preprod_payload_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", preprod_payload_data_set_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_response = wml_client.deployments.score(pre_prod_deployment_uid, payload_scoring)\n",
    "\n",
    "print(\"Single record scoring result:\", \"\\n fields:\", scoring_response[\"predictions\"][0][\"fields\"], \"\\n values: \", scoring_response[\"predictions\"][0][\"values\"][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
    "time.sleep(5)\n",
    "pl_records_count = wos_client.data_sets.get_records_count(preprod_payload_data_set_id)\n",
    "print(\"Number of records in the payload logging table: {}\".format(pl_records_count))\n",
    "if pl_records_count == 0:\n",
    "    print(\"Payload logging did not happen, performing explicit payload logging.\")\n",
    "    wos_client.data_sets.store_records(data_set_id=preprod_payload_data_set_id, request_body=[PayloadRecord(\n",
    "                   scoring_id=str(uuid.uuid4()),\n",
    "                   request=payload_scoring,\n",
    "                   response=scoring_response,\n",
    "                   response_time=460\n",
    "               )])\n",
    "    time.sleep(5)\n",
    "    pl_records_count = wos_client.data_sets.get_records_count(preprod_payload_data_set_id)\n",
    "    print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(5)\n",
    "challenger_payload_data_set_id = None\n",
    "challenger_payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=challenger_subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if challenger_payload_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", challenger_payload_data_set_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_response = wml_client.deployments.score(challenger_deployment_uid, payload_scoring)\n",
    "\n",
    "print(\"Single record scoring result:\", \"\\n fields:\", scoring_response[\"predictions\"][0][\"fields\"], \"\\n values: \", scoring_response[\"predictions\"][0][\"values\"][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n",
    "time.sleep(5)\n",
    "pl_records_count = wos_client.data_sets.get_records_count(challenger_payload_data_set_id)\n",
    "print(\"Number of records in the payload logging table: {}\".format(pl_records_count))\n",
    "if pl_records_count == 0:\n",
    "    print(\"Payload logging did not happen, performing explicit payload logging.\")\n",
    "    wos_client.data_sets.store_records(data_set_id=challenger_payload_data_set_id, request_body=[PayloadRecord(\n",
    "                   scoring_id=str(uuid.uuid4()),\n",
    "                   request=payload_scoring,\n",
    "                   response=scoring_response,\n",
    "                   response_time=460\n",
    "               )])\n",
    "    time.sleep(5)\n",
    "    pl_records_count = wos_client.data_sets.get_records_count(challenger_payload_data_set_id)\n",
    "    print(\"Number of records in the payload logging table: {}\".format(pl_records_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quality monitoring\n",
    "\n",
    "## Enable quality monitoring\n",
    "The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 80%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.\n",
    "\n",
    "The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(10)\n",
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=pre_prod_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"min_feedback_data_size\": 50\n",
    "}\n",
    "quality_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "preprod_quality_monitor_instance_id = quality_monitor_details.metadata.id\n",
    "preprod_quality_monitor_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(10)\n",
    "target = Target(\n",
    "        target_type=TargetTypes.SUBSCRIPTION,\n",
    "        target_id=challenger_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"min_feedback_data_size\": 50\n",
    "}\n",
    "quality_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "challenger_quality_monitor_instance_id = quality_monitor_details.metadata.id\n",
    "challenger_quality_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fairness, drift monitoring and explanations \n",
    "\n",
    "## Fairness configuration\n",
    "The code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:\n",
    "\n",
    "Which model feature to monitor\n",
    "One or more majority groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes\n",
    "One or more minority groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes\n",
    "The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 80%)\n",
    "Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 100 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"features\": [\n",
    "        {\"feature\": \"Sex\",\n",
    "         \"majority\": ['male'],\n",
    "         \"minority\": ['female'],\n",
    "         \"threshold\": 0.95\n",
    "         },\n",
    "        {\"feature\": \"Age\",\n",
    "         \"majority\": [[26, 75]],\n",
    "         \"minority\": [[18, 25]],\n",
    "         \"threshold\": 0.95\n",
    "         }\n",
    "    ],\n",
    "    \"favourable_class\": [\"No Risk\"],\n",
    "    \"unfavourable_class\": [\"Risk\"],\n",
    "    \"min_records\": 100\n",
    "}\n",
    "\n",
    "fairness_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
    "    target=target,\n",
    "    parameters=parameters).result\n",
    "preprod_fairness_monitor_instance_id =fairness_monitor_details.metadata.id\n",
    "preprod_fairness_monitor_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"features\": [\n",
    "        {\"feature\": \"Sex\",\n",
    "         \"majority\": ['male'],\n",
    "         \"minority\": ['female'],\n",
    "         \"threshold\": 0.95\n",
    "         },\n",
    "        {\"feature\": \"Age\",\n",
    "         \"majority\": [[26, 75]],\n",
    "         \"minority\": [[18, 25]],\n",
    "         \"threshold\": 0.95\n",
    "         }\n",
    "    ],\n",
    "    \"favourable_class\": [\"No Risk\"],\n",
    "    \"unfavourable_class\": [\"Risk\"],\n",
    "    \"min_records\": 100\n",
    "}\n",
    "\n",
    "fairness_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,\n",
    "    target=target,\n",
    "    parameters=parameters).result\n",
    "challenger_fairness_monitor_instance_id =fairness_monitor_details.metadata.id\n",
    "challenger_fairness_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drift configuration\n",
    "\n",
    "Enable the drift configuration for both the subscription created with a threshold of 10% and minimal sample as 100 records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"min_samples\": 100,\n",
    "    \"drift_threshold\": 0.1,\n",
    "    \"train_drift_model\": True,\n",
    "    \"enable_model_drift\": False,\n",
    "    \"enable_data_drift\": True\n",
    "}\n",
    "\n",
    "drift_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "preprod_drift_monitor_instance_id = drift_monitor_details.metadata.id\n",
    "preprod_drift_monitor_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    "\n",
    ")\n",
    "parameters = {\n",
    "    \"min_samples\": 100,\n",
    "    \"drift_threshold\": 0.1,\n",
    "    \"train_drift_model\": True,\n",
    "    \"enable_model_drift\": False,\n",
    "    \"enable_data_drift\": True\n",
    "}\n",
    "\n",
    "drift_v2_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "challenger_preprod_drift_monitor_instance_id = drift_monitor_details.metadata.id\n",
    "challenger_preprod_drift_monitor_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drift V2 Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import choices\n",
    "\n",
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    ")\n",
    "\n",
    "fields = [\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\n",
    "\n",
    "parameters = {\n",
    "        \"min_samples\": 10,\n",
    "        \"max_samples\": 1000,\n",
    "        \"train_archive\": True,\n",
    "        \"features\": {\n",
    "            \"fields\": fields,\n",
    "        },\n",
    "        \"most_important_features\": { \n",
    "            \"fields\": choices(fields, k=5)\n",
    "        },\n",
    "    }\n",
    "drift_v2_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT_V2.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "preprod_drift_v2_monitor_instance_id = drift_v2_monitor_details.metadata.id\n",
    "preprod_drift_v2_monitor_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    ")\n",
    "\n",
    "parameters = {\n",
    "        \"min_samples\": 10,\n",
    "        \"max_samples\": 1000,\n",
    "        \"train_archive\": True,\n",
    "        \"features\": {\n",
    "            \"fields\": fields,\n",
    "        },\n",
    "        \"most_important_features\": { \n",
    "            \"fields\": choices(fields, k=5)\n",
    "        },\n",
    "    }\n",
    "drift_v2_monitor_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT_V2.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "challenger_drift_v2_monitor_instance_id = drift_v2_monitor_details.metadata.id\n",
    "challenger_drift_v2_monitor_instance_id\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Explainability\n",
    "Finally, we provide OpenScale with the training data to enable and configure the explainability features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"enabled\": True\n",
    "}\n",
    "explainability_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "preprod_explainability_monitor_id = explainability_details.metadata.id\n",
    "preprod_explainability_monitor_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    ")\n",
    "parameters = {\n",
    "    \"enabled\": True\n",
    "}\n",
    "explainability_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    ").result\n",
    "\n",
    "challenger_explainability_monitor_id = explainability_details.metadata.id\n",
    "challenger_explainability_monitor_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable model risk management (MRM) \n",
    "\n",
    "We enable the MRM configuration for both the subscriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=pre_prod_subscription_id\n",
    ")\n",
    "\n",
    "parameters = {}\n",
    "\n",
    "mrm_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.MODEL_RISK_MANAGEMENT_MONITORING.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    "    \n",
    ").result\n",
    "\n",
    "\n",
    "pre_prod_mrm_instance_id = mrm_details.metadata.id\n",
    "pre_prod_mrm_instance_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = Target(\n",
    "    target_type=TargetTypes.SUBSCRIPTION,\n",
    "    target_id=challenger_subscription_id\n",
    ")\n",
    "\n",
    "parameters = {}\n",
    "\n",
    "challenger_mrm_details = wos_client.monitor_instances.create(\n",
    "    data_mart_id=data_mart_id,\n",
    "    background_mode=False,\n",
    "    monitor_definition_id=wos_client.monitor_definitions.MONITORS.MODEL_RISK_MANAGEMENT_MONITORING.ID,\n",
    "    target=target,\n",
    "    parameters=parameters\n",
    "    \n",
    ").result\n",
    "\n",
    "\n",
    "challenger_mrm_instance_id = challenger_mrm_details.metadata.id\n",
    "challenger_mrm_instance_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Patch the integration reference in the pre-prod subscription"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "payload = [\n",
    "  {\n",
    "    \"op\": \"add\",\n",
    "    \"path\": \"/integration_reference\",\n",
    "    \"value\": {\n",
    "        \"integrated_system_id\": integrated_system_id,\n",
    "        \"external_id\": openpages_model_id\n",
    "    }\n",
    "  }\n",
    "]\n",
    "\n",
    "SUBSCRIPTIONS_URL =\"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/subscriptions/{1}\".format(WOS_GUID, pre_prod_subscription_id)\n",
    "\n",
    "response = requests.patch(SUBSCRIPTIONS_URL, json=payload, headers=headers)\n",
    "json_data = response.json()\n",
    "json_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create test data sets from the training data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data_1 = pd_data[1:201]\n",
    "test_data_1.to_csv(\"german_credit_risk_test_data_1.csv\", encoding=\"utf-8\", index=False)\n",
    "test_data_2 = pd_data[201:401]\n",
    "test_data_2.to_csv(\"german_credit_risk_test_data_2.csv\", encoding=\"utf-8\", index=False)\n",
    "test_data_3 = pd_data[401:601]\n",
    "test_data_3.to_csv(\"german_credit_risk_test_data_3.csv\", encoding=\"utf-8\", index=False)\n",
    "test_data_4 = pd_data[601:801]\n",
    "test_data_4.to_csv(\"german_credit_risk_test_data_4.csv\", encoding=\"utf-8\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to upload, evaluate and check the status of the evaluation\n",
    "\n",
    "This function will upload the test data CSV and trigger the risk evaluation. It will iterate and check the status of the evaluation until its finished with a finite wait duration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def upload_and_evaluate(file_name, mrm_instance_id):\n",
    "    \n",
    "    print(\"Running upload and evaluate for {}\".format(file_name))\n",
    "    import time\n",
    "    from datetime import datetime\n",
    "\n",
    "    status = None\n",
    "    monitoring_run_id = None\n",
    "    GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES = 32\n",
    "    GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL = 10\n",
    "    \n",
    "    if file_name is not None:\n",
    "        \n",
    "        response = wos_client.monitor_instances.mrm.evaluate_risk(\n",
    "                        monitor_instance_id = mrm_instance_id,\n",
    "                        test_data_set_name = file_name,\n",
    "                        feedback_data_path = file_name)\n",
    "        \n",
    "        if response.status_code != 202:\n",
    "            print(\"Upload and evalaute for {0} failed with error : {1}\".format(file_name, response.status_code))\n",
    "            return\n",
    "        \n",
    "        for i in range(GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES):\n",
    "        \n",
    "            response = wos_client.monitor_instances.mrm.get_risk_evaluation(\n",
    "                            monitor_instance_id = mrm_instance_id)\n",
    "\n",
    "            if response.status_code != 200:\n",
    "                print(\"Getting status of upload and evalaute for {0} failed with error : {1}\".format(file_name, response.status_code))\n",
    "                return\n",
    "      \n",
    "            response = response.result._to_dict()            \n",
    "            \n",
    "            if \"metadata\" in response and \"id\" in response[\"metadata\"]:\n",
    "                monitoring_run_id = response[\"metadata\"][\"id\"]\n",
    "            if \"entity\" in response and \"status\" in response[\"entity\"]:\n",
    "                status = response[\"entity\"][\"status\"][\"state\"]\n",
    "            \n",
    "            if status is not None:\n",
    "                print(datetime.utcnow().strftime('%H:%M:%S'), status.lower())\n",
    "                if status.lower() in [\"finished\", \"completed\"]:\n",
    "                    break\n",
    "                elif \"error\"in status.lower():\n",
    "                    print(response)\n",
    "                    break\n",
    "\n",
    "            time.sleep(GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL)\n",
    "\n",
    "    return status, monitoring_run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform Risk Evaluations\n",
    "\n",
    "We now start performing evaluations of smaller data sets against both the PreProd and Challenger subscriptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_1.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_2.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_3.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_4.csv\", pre_prod_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_1.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_2.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_3.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_and_evaluate(\"german_credit_risk_test_data_4.csv\", challenger_mrm_instance_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Model Risk Management UI\n",
    "\n",
    "Here is a quick recap of what we have done so far.\n",
    "\n",
    "1. We've deployed two Credit Risk Model to a WML instance that is designated as Pre-Production\n",
    "2. We've created subscriptions of these two model deployments in OpenScale\n",
    "3. Configured all monitors supported by OpenScale for these subscriptions\n",
    "4. We've performed a few risk evaluations against both these susbscription with the same set of test data\n",
    "\n",
    "Now, please explore the Model Risk Management UI to visualize the results, compare the performance of models, download the evaluation report as PDF. For more information, refer to the Beta Guide section \"Work in Watson OpenScale.\"\n",
    "\n",
    "Link to OpenScale : https://aiopenscale.cloud.ibm.com/aiopenscale/insights?mrm=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Promote pre-production model to production \n",
    "\n",
    "After you have reviewed the evaluation results of the PreProd Vs Challenger and if you make the decision to promote the PreProd model to Production, the first thing you need to do is to deploy the model into a WML instance that is designated as Production instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy model to production WML instance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROD_MODEL_NAME=\"German Credit Risk Model - Prod\"\n",
    "PROD_DEPLOYMENT_NAME=\"German Credit Risk Model - Prod\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "space_name =\"prod-space\"\n",
    "spaces = wml_client.spaces.get_details()['resources']\n",
    "prod_space_id = None\n",
    "for space in spaces:\n",
    "    if space['entity']['name'] == space_name:\n",
    "        prod_space_id = space[\"metadata\"][\"id\"]\n",
    "if prod_space_id is None:\n",
    "    prod_space_id = wml_client.spaces.store(\n",
    "        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name,\n",
    "                   wml_client.spaces.ConfigurationMetaNames.STORAGE: {\"resource_crn\":COS_RESOURCE_CRN},\n",
    "                   wml_client.spaces.ConfigurationMetaNames.COMPUTE: {\"name\": WML_INSTANCE_NAME,\n",
    "                                            \"crn\": WML_CRN}})[\"metadata\"][\"id\"]\n",
    "wml_client.set.default_space(prod_space_id)\n",
    "print(prod_space_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prod_model_uid, prod_deployment_uid, prod_scoring_url = deploy_credit_risk_spark_model(WML_CREDENTIALS, PROD_MODEL_NAME, PROD_DEPLOYMENT_NAME,prod_space_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bind WML machine learning instance as Prod\n",
    "\n",
    "Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name \"WML Prod\" already exists, this code will delete that binding a create a new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SERVICE_PROVIDER_NAME = \"Watson Machine Learning prod openpage\"\n",
    "SERVICE_PROVIDER_DESCRIPTION = \"Added by tutorial WOS notebook.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_providers = wos_client.service_providers.list().result.service_providers\n",
    "for service_provider in service_providers:\n",
    "    service_instance_name = service_provider.entity.name\n",
    "    if service_instance_name == SERVICE_PROVIDER_NAME:\n",
    "        service_provider_id = service_provider.metadata.id\n",
    "        wos_client.service_providers.delete(service_provider_id)\n",
    "        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "added_service_provider_result = wos_client.service_providers.add(\n",
    "        name=SERVICE_PROVIDER_NAME,\n",
    "        description=SERVICE_PROVIDER_DESCRIPTION,\n",
    "        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,\n",
    "        deployment_space_id = prod_space_id, # use prod space ID\n",
    "        operational_space_id = \"production\",\n",
    "        credentials=WMLCredentialsCloud(\n",
    "            apikey=CLOUD_API_KEY,      ## use `apikey=IAM_TOKEN` if using IAM_TOKEN to initiate client\n",
    "            url=WML_CREDENTIALS[\"url\"],\n",
    "            instance_id=None\n",
    "        ),\n",
    "        background_mode=False\n",
    "    ).result\n",
    "service_provider_id = added_service_provider_result.metadata.id\n",
    "service_provider_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id, deployment_space_id = prod_space_id).result['resources'][0]\n",
    "asset_deployment_details"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import configuration settings from pre-prod model\n",
    "\n",
    "With MRM we provide a important feature that lets you copy the configuration settings of your pre-production subscription to the production subscription. To try this out\n",
    "\n",
    "1. Navigate to Model Monitors view in Insights dashboard of OpenScale\n",
    "2. Click on the Add to dashboard\n",
    "3. Select the production model deployment from WML production machine learning provider and click on Configure\n",
    "4. In Selections saved dialog, click on Configure monitors\n",
    "4. In Model Governance, click on Begin and choose the same OpenPages model that was linked to the pre-production model and click on Save\n",
    "5. You will be presented with an dialog about the OpenPages model being associated with another subscription and if you'd like to import settings.\n",
    "6. Click on Import and confirm\n",
    "\n",
    "All the configuration settings are now copied into the production subscription\n",
    "\n",
    "\n",
    "<b>Note: The next set of cells should be executed only after finishing the import settings from the OpenScale dashboard</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score the production model so that we can trigger monitors\n",
    "\n",
    "Now that the production subscription is configured by copying the configuration, there would be schedules created for each of the monitors to run on a scheduled basis. \n",
    "Quality, Fairness and Mrm will run hourly. Drift will run once in three hours.\n",
    "\n",
    "For this demo purpose, we will trigger the monitors on-demand so that we can see the model summary dashboard without having to wait the entire hour. \n",
    "To do that lets first push some records in the Payload Logging table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd_data.sample(n=400)\n",
    "df = df.drop(['Risk'], axis=1)\n",
    "fields = df.columns.tolist()\n",
    "values = df.values.tolist()\n",
    "payload_scoring = {\"input_data\": [{\"fields\": fields, \"values\": values}]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_response = wml_client.deployments.score(prod_deployment_uid, payload_scoring)\n",
    "print(\"Single record scoring result:\", \"\\n fields:\", scoring_response[\"predictions\"][0][\"fields\"], \"\\n values: \", scoring_response[\"predictions\"][0][\"values\"][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wos_client.subscriptions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prod_subscription = wos_client.subscriptions.get('5949851d-8746-4bfa-b3f0-9a27c4cef98b').result.to_dict()\n",
    "prod_subscription_id=prod_subscription['metadata']['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "time.sleep(5)\n",
    "prod_payload_data_set_id = None\n",
    "prod_payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n",
    "                                                target_target_id=prod_subscription_id, \n",
    "                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\n",
    "if prod_payload_data_set_id is None:\n",
    "    print(\"Payload data set not found. Please check subscription status.\")\n",
    "else:\n",
    "    print(\"Payload data set id: \", prod_payload_data_set_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetch all monitor instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "MONITOR_INSTANCES_URL = \"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/monitor_instances?target.target_id={1}&target.target_type=subscription\".format(WOS_GUID, prod_subscription_id)\n",
    "print(MONITOR_INSTANCES_URL)\n",
    "\n",
    "response = requests.get(MONITOR_INSTANCES_URL, headers=headers)\n",
    "monitor_instances = response.json()[\"monitor_instances\"]\n",
    "\n",
    "drift_monitor_instance_id = None\n",
    "quality_monitor_instance_id = None\n",
    "fairness_monitor_instance_id= None\n",
    "mrm_monitor_instance_id = None\n",
    "\n",
    "if monitor_instances is not None:\n",
    "    for monitor_instance in monitor_instances:\n",
    "        if \"entity\" in monitor_instance and \"monitor_definition_id\" in monitor_instance[\"entity\"]:\n",
    "            monitor_name = monitor_instance[\"entity\"][\"monitor_definition_id\"]\n",
    "            if \"metadata\" in monitor_instance and \"id\" in monitor_instance[\"metadata\"]:\n",
    "                id = monitor_instance[\"metadata\"][\"id\"]\n",
    "                if monitor_name == \"drift\":\n",
    "                    drift_monitor_instance_id = id\n",
    "                elif monitor_name == \"drift_v2\":\n",
    "                    drift_v2_monitor_instance_id = id\n",
    "                elif monitor_name == \"fairness\":\n",
    "                    fairness_monitor_instance_id = id\n",
    "                elif monitor_name == \"quality\":\n",
    "                    quality_monitor_instance_id = id\n",
    "                elif monitor_name == \"mrm\":\n",
    "                    mrm_monitor_instance_id = id\n",
    "                    \n",
    "print(\"Quality monitor instance id - {0}\".format(quality_monitor_instance_id))\n",
    "print(\"Fairness monitor instance id - {0}\".format(fairness_monitor_instance_id))\n",
    "print(\"Drift monitor instance id - {0}\".format(drift_monitor_instance_id))\n",
    "print(\"Drift v2 monitor instance id - {0}\".format(drift_v2_monitor_instance_id))\n",
    "print(\"MRM monitor instance id - {0}\".format(mrm_monitor_instance_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function to get the monitoring run details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_monitoring_run_details(monitor_instance_id, monitoring_run_id):\n",
    "    \n",
    "    headers = {}\n",
    "    headers[\"Content-Type\"] = \"application/json\"\n",
    "    headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "    \n",
    "    MONITORING_RUNS_URL = \"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/monitor_instances/{1}/runs/{2}\".format(WOS_GUID, monitor_instance_id, monitoring_run_id)\n",
    "    response = requests.get(MONITORING_RUNS_URL, headers=headers, verify=False)\n",
    "    return response.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on-demand Quality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "if quality_monitor_instance_id is not None:\n",
    "    MONITOR_RUN_URL = \"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, quality_monitor_instance_id)\n",
    "    payload = {\n",
    "        \"triggered_by\": \"user\"\n",
    "    }\n",
    "    print(\"Triggering Quality computation with {}\".format(MONITOR_RUN_URL))\n",
    "    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n",
    "    json_data = response.json()\n",
    "    print()\n",
    "    print(json_data)\n",
    "    print()\n",
    "    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n",
    "        quality_monitoring_run_id = json_data[\"metadata\"][\"id\"]\n",
    "    print(\"Done triggering Quality computation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "quality_run_status = None\n",
    "while quality_run_status != 'finished':\n",
    "    monitoring_run_details = get_monitoring_run_details(quality_monitor_instance_id, quality_monitoring_run_id)\n",
    "    quality_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n",
    "    if quality_run_status == \"error\":\n",
    "        print(monitoring_run_details)\n",
    "        break\n",
    "    if quality_run_status != 'finished':\n",
    "        print(datetime.utcnow().strftime('%H:%M:%S'), quality_run_status)\n",
    "        time.sleep(10)\n",
    "print(quality_run_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on-demand Drift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "if drift_monitor_instance_id is not None:\n",
    "    MONITOR_RUN_URL = \"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, drift_monitor_instance_id)\n",
    "    payload = {\n",
    "        \"triggered_by\": \"user\"\n",
    "    }\n",
    "    print(\"Triggering Drift computation with {}\".format(MONITOR_RUN_URL))\n",
    "    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n",
    "    json_data = response.json()\n",
    "    print()\n",
    "    print(json_data)\n",
    "    print()\n",
    "    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n",
    "        drift_monitoring_run_id = json_data[\"metadata\"][\"id\"]\n",
    "    print(\"Done triggering Drift computation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "drift_run_status = None\n",
    "while drift_run_status != 'finished':\n",
    "    monitoring_run_details = get_monitoring_run_details(drift_monitor_instance_id, drift_monitoring_run_id)\n",
    "    drift_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n",
    "    if drift_run_status == \"error\":\n",
    "        print(monitoring_run_details)\n",
    "        break\n",
    "    if drift_run_status != 'finished':\n",
    "        print(datetime.utcnow().strftime('%H:%M:%S'), drift_run_status)\n",
    "        time.sleep(10)\n",
    "print(drift_run_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on-demand Drift V2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "if drift_v2_monitor_instance_id is not None:\n",
    "    MONITOR_RUN_URL = \"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, drift_v2_monitor_instance_id)\n",
    "    payload = {\n",
    "        \"triggered_by\": \"user\"\n",
    "    }\n",
    "    print(\"Triggering Drift V2 computation with {}\".format(MONITOR_RUN_URL))\n",
    "    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n",
    "    json_data = response.json()\n",
    "    print()\n",
    "    print(json_data)\n",
    "    print()\n",
    "    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n",
    "        drift_v2_monitoring_run_id = json_data[\"metadata\"][\"id\"]\n",
    "    print(\"Done triggering Drift V2 computation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "drift_v2_run_status = None\n",
    "while drift_v2_run_status != 'finished':\n",
    "    monitoring_run_details = get_monitoring_run_details(drift_v2_monitor_instance_id, drift_v2_monitoring_run_id)\n",
    "    drift_v2_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n",
    "    if drift_v2_run_status == \"error\":\n",
    "        print(monitoring_run_details)\n",
    "        break\n",
    "    if drift_v2_run_status != 'finished':\n",
    "        print(datetime.utcnow().strftime('%H:%M:%S'), drift_run_status)\n",
    "        time.sleep(10)\n",
    "print(drift_v2_run_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on-demand Fairness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "if fairness_monitor_instance_id is not None:\n",
    "    MONITOR_RUN_URL = \"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, fairness_monitor_instance_id)\n",
    "    payload = {\n",
    "        \"triggered_by\": \"user\"\n",
    "    }\n",
    "    print(\"Triggering fairness computation with {}\".format(MONITOR_RUN_URL))\n",
    "    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n",
    "    json_data = response.json()\n",
    "    print()\n",
    "    print(json_data)\n",
    "    print()\n",
    "    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n",
    "        fairness_monitor_run_id = json_data[\"metadata\"][\"id\"]\n",
    "    print(\"Done triggering fairness computation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "fairness_run_status = None\n",
    "while fairness_run_status != 'finished':\n",
    "    monitoring_run_details = get_monitoring_run_details(fairness_monitor_instance_id, fairness_monitor_run_id)\n",
    "    fairness_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n",
    "    if fairness_run_status == \"error\":\n",
    "        print(monitoring_run_details)\n",
    "        break\n",
    "    if fairness_run_status != 'finished':\n",
    "        print(datetime.utcnow().strftime('%H:%M:%S'), fairness_run_status)\n",
    "        time.sleep(10)\n",
    "print(fairness_run_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run on-demand MRM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "headers = {}\n",
    "headers[\"Content-Type\"] = \"application/json\"\n",
    "headers[\"Authorization\"] = \"Bearer {}\".format(generate_access_token())\n",
    "\n",
    "if mrm_monitor_instance_id is not None:\n",
    "    MONITOR_RUN_URL =\"https://api.aiopenscale.cloud.ibm.com/openscale/{0}/v2/monitor_instances/{1}/runs\".format(WOS_GUID, mrm_monitor_instance_id)\n",
    "    payload = {\n",
    "        \"triggered_by\": \"user\"\n",
    "    }\n",
    "    print(\"Triggering MRM computation with {}\".format(MONITOR_RUN_URL))\n",
    "    response = requests.post(MONITOR_RUN_URL, json=payload, headers=headers, verify=False)\n",
    "    json_data = response.json()\n",
    "    print()\n",
    "    print(json_data)\n",
    "    print()\n",
    "    if \"metadata\" in json_data and \"id\" in json_data[\"metadata\"]:\n",
    "        mrm_monitoring_run_id = json_data[\"metadata\"][\"id\"]\n",
    "    print(\"Done triggering MRM computation\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "mrm_run_status = None\n",
    "while mrm_run_status != 'finished':\n",
    "    monitoring_run_details = get_monitoring_run_details(mrm_monitor_instance_id, mrm_monitoring_run_id)\n",
    "    mrm_run_status = monitoring_run_details[\"entity\"][\"status\"][\"state\"]\n",
    "    if mrm_run_status == \"error\":\n",
    "        print(monitoring_run_details)\n",
    "        break\n",
    "    if mrm_run_status != 'finished':\n",
    "        print(datetime.utcnow().strftime('%H:%M:%S'), mrm_run_status)\n",
    "        time.sleep(10)\n",
    "print(mrm_run_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Refresh the model summary of the production subscription in the OpenScale dashboard\n",
    "\n",
    "This brings us to the end of this demo exercise. Thank you for trying it out."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}