{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7772f52c-2d12-4b71-bf03-e5f8b64e8eb3",
   "metadata": {},
   "source": [
    "# Managing a CryoSPARC Live Session from the CLI\n",
    "This guide details how to automate the creation, configuration and overall management of a [CryoSPARC Live](https://guide.cryosparc.com/live/about-cryosparc-live) session in Python via the CryoSPARC API.\n",
    "```{note}\n",
    "The commands on this page apply to CryoSPARC v5.\n",
    "```\n",
    "\n",
    "## Setup\n",
    "### CryoSPARC Connection and Tools API Object\n",
    "\n",
    "Specify the master hostname and port number of the CryoSPARC installation where CryoSPARC Live will run. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e94f2cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "master_hostname = \"localhost\"  # should match hostname portion of login token\n",
    "base_port = 62000  # should match port portion of login token"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1ca911a",
   "metadata": {},
   "source": [
    "Connect to the CryoSPARC instance and define the api object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83a342f0-ee30-4858-a2a1-7db09051e557",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "import cryosparc.tools\n",
    "\n",
    "cs = cryosparc.tools.CryoSPARC(f\"http://{master_hostname}:{base_port}\")\n",
    "assert cs.test_connection()\n",
    "api = cs.api"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5debf88-cf7e-4fb4-b3c7-1a73f21e9fb2",
   "metadata": {},
   "source": [
    "### Available Parameters\n",
    "The full list of session parameters that can be modified can be obtained with the command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b9def92-5754-4d13-b9c1-6ef72ac2b2d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.sessions.get_session_base_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90ed8dfb-8ffa-4658-99c6-2f2bc32c7ba2",
   "metadata": {},
   "source": [
    "### Raw Data Specification and Processing Parameters\n",
    "If one uses an existing CryoSPARC project, one does not need to define the `project_parent_dir`, `project_title` and `project_description` variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9923e8d2-6a46-473b-8b88-43fda2096d37",
   "metadata": {},
   "outputs": [],
   "source": [
    "project_parent_dir = \"/bulk2/CS/prod_projects\"  # where to create CryoSPARC project dir\n",
    "\n",
    "project_title = \"Live test\"\n",
    "\n",
    "project_description = \"Live processing of the EMPIAR-10025 subset\"\n",
    "\n",
    "session_title = \"Live test session\"\n",
    "\n",
    "session_description = \"realtime processing of EMPIAR-10025 subset\"\n",
    "\n",
    "exposure_group_configs = [\n",
    "    {\n",
    "        \"file_engine_watch_path_abs\": \"/bulk5/data/empiar_10025_subset/\",\n",
    "        \"file_engine_filter\": \"14sep05c_00024sq_*.frames.tif\",\n",
    "        \"gainref_path\": \"/bulk5/data/empiar_10025_subset/norm-amibox05-0.mrc\",\n",
    "    },\n",
    "    {\n",
    "        \"file_engine_watch_path_abs\": \"/bulk5/data/empiar_10025_subset/\",\n",
    "        \"file_engine_filter\": \"14sep05c_c_00003gr_00014sq_*.frames.tif\",\n",
    "        \"gainref_path\": \"/bulk5/data/empiar_10025_subset/norm-amibox05-0.mrc\",\n",
    "    },\n",
    "]\n",
    "\n",
    "session_params = {\n",
    "    \"psize_A\": 0.6575,\n",
    "    \"accel_kv\": 300,\n",
    "    \"cs_mm\": 2.7,\n",
    "    \"total_dose_e_per_A2\": 53,\n",
    "    \"blob_diameter_min\": 100,\n",
    "    \"blob_diameter_max\": 200,\n",
    "    \"box_size_pix\": 440,\n",
    "    \"bin_size_pix\": 256,\n",
    "    \"output_f16\": True,\n",
    "    \"extract_f16\": True,\n",
    "}\n",
    "\n",
    "class2D_params = {\"class2D_K\": 20}\n",
    "\n",
    "# for ab initio 3D reconstruction\n",
    "abinit_params = {\"abinit_K\": 1, \"abinit_num_particles\": 9000}\n",
    "\n",
    "refine_params = {\"refine_symmetry\": \"D7\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7f5d40a-c41d-4265-9c39-bc007a003185",
   "metadata": {},
   "source": [
    "### Compute Resources\n",
    "Specify which scheduler lanes on your CryoSPARC instance to use for various processing tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8269ab93-f308-4dbd-bd99-5383437245ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "compute_resources = {\n",
    "    \"phase_one_lane\": \"slurm_4\",  # for motion correction thru particle extraction\n",
    "    \"phase_one_gpus\": 1,\n",
    "    \"phase_two_lane\": \"slurm_4\",  # streaming 2D classification and 3D refinement\n",
    "    \"phase_two_ssd\": False,  # set True for particle caching\n",
    "    \"auxiliary_lane\": \"slurm_4\",  # transient or optional jobs, like 3d ab initio reconstruction\n",
    "    \"auxiliary_ssd\": False,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "171c2901-371e-4d18-9632-d9ed8df28b30",
   "metadata": {},
   "source": [
    "### Create CryoSPARC Project or Specify Existing Project's UID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c7a8ae4-4da4-4299-8555-acb932681eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "project_uid = \"P291\"  # existing project"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35c9fdb2-a11b-4d52-91ec-9192fa177fe6",
   "metadata": {},
   "source": [
    "To use an existing project, keep the `project_uid = \"P291\"` line above. To create a new project instead, comment out that line and uncomment the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "564984e6-d279-44c7-b3b3-5f8ac83ccc75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# project_uid = cs.api.projects.create(title=project_title,\n",
    "#                                     description=project_description,\n",
    "#                                     parent_dir=project_parent_dir).uid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d19934d1-3477-45aa-8b7b-f64b849f6924",
   "metadata": {},
   "source": [
    "## Create Session and Capture Session ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5ce8c7e-75d1-40cc-a73e-7a7074b6823d",
   "metadata": {},
   "outputs": [],
   "source": [
    "session_uid = api.sessions.create(project_uid, title=session_title, description=session_description).session_uid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a2e31fc-fafe-4f3f-8e70-13e7468d84fc",
   "metadata": {},
   "source": [
    "### Optional Automatic Start and Pause\n",
    "\n",
    "CryoSPARC Live can wait until exposures are available on the filesystem before starting the session. This prevents the session from using hardware resources or license tokens until raw movie files are available for processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56d5349c-c430-4fa4-b12e-8934841078b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.sessions.set_session_phase_one_wait_for_exposures(project_uid, session_uid, phase_one_wait_for_exposures=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff99aeb9-4e02-4146-a1f5-1483701524c6",
   "metadata": {},
   "source": [
    "A session can be paused automatically when it has become idle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d233b74-530d-4b07-9f8a-b353637a5844",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.sessions.configure_auto_pause(project_uid, session_uid, auto_pause=\"graceful\", auto_pause_after_idle_minutes=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ada4b3c4-4d75-4e7f-899d-be1ac2126610",
   "metadata": {},
   "source": [
    "## Configuration and Start of Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4fc6199-9ed2-4f5b-8ddc-73827e4cb1f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.sessions.update_compute_configuration(project_uid, session_uid, compute_resources)\n",
    "\n",
    "# one exposure group may already exist in the newly created session\n",
    "for i in range(len(exposure_group_configs) - len(api.sessions.find_exposure_groups(project_uid, session_uid))):\n",
    "    api.sessions.create_exposure_group(project_uid, session_uid)\n",
    "\n",
    "for eg in zip(api.sessions.find_exposure_groups(project_uid, session_uid), exposure_group_configs):\n",
    "    api.sessions.update_exposure_group(project_uid, session_uid, eg[0].exp_group_id, eg[1])\n",
    "    api.sessions.finalize_exposure_group(project_uid, session_uid, eg[0].exp_group_id)\n",
    "\n",
    "api.sessions.update_session_params(project_uid, session_uid, session_params)\n",
    "\n",
    "api.sessions.start(project_uid, session_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e13d77f1-38d1-40a1-bcdd-114299b7e4d2",
   "metadata": {},
   "source": [
    "## Start Streaming 2D Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4898bdee-48a0-437a-9538-6a9c0122822e",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.sessions.update_phase2_class2D_params(project_uid, session_uid, class2D_params)\n",
    "api.sessions.setup_phase2_class2D(project_uid, session_uid)\n",
    "api.sessions.enqueue_phase2_class2D(project_uid, session_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2e95235-4e2a-44b0-a174-550a62d96eb5",
   "metadata": {},
   "source": [
    "## Select 2D Classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d01d2dae-ab44-4e07-b1fa-f7e469d5b09a",
   "metadata": {},
   "outputs": [],
   "source": [
    "while (\n",
    "    api.sessions.find_one(project_uid, session_uid).phase2_class2D_num_particles_seen\n",
    "    < abinit_params[\"abinit_num_particles\"]\n",
    "):\n",
    "    time.sleep(10)\n",
    "\n",
    "api.sessions.select_all_class2d_templates(project_uid, session_uid, \"select\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9e183a5-099a-4253-a75d-503a51db3c05",
   "metadata": {},
   "source": [
    "In this example, all templates (and associated particles) are selected. In a production scenario, one may select a meaningful subset of templates using one of the [`select_class2d_templates_*`](https://tools.cryosparc.com/api/cryosparc.api.html#cryosparc.api.SessionsAPI.select_class2d_templates_with_threshold_index) functions.\n",
    "## Start _ab initio_ 3D Reconstruction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77fec60e-50f7-4b03-b465-3dedcb96dc0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.sessions.update_phase2_abinit_params(project_uid, session_uid, abinit_params)\n",
    "abinit_job_uid = api.sessions.setup_phase2_abinit(project_uid, session_uid).uid\n",
    "api.sessions.enqueue_phase2_abinit(project_uid, session_uid)\n",
    "cs.find_job(project_uid, abinit_job_uid).wait_for_status(\"completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "121c564c-a8d6-402b-b925-ac8b1f248266",
   "metadata": {},
   "source": [
    "and wait for its completion.\n",
    "## Start Streaming 3D Refinement\n",
    "... using the first (0-based index) volume as input. `volume_class_0` is a plausible choice if there was a single class (`abinit_K: 1`) _ab initio_ 3D reconstruction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eae2f975-6c71-4aeb-b6fd-8542abbb53d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "api.sessions.select_phase2_abinit_volume(project_uid, session_uid, volume_name=\"volume_class_0\")\n",
    "api.sessions.update_phase2_refine_params(project_uid, session_uid, refine_params)\n",
    "refine_job_uid = api.sessions.setup_phase2_refine(project_uid, session_uid).uid\n",
    "api.sessions.enqueue_phase2_refine(project_uid, session_uid)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "697483a3-d23e-44ac-b326-95719edd387c",
   "metadata": {},
   "source": [
    "## Export Exposures and Particles\n",
    "... for downstream processing. In this example, the export is preceded by periodic checks whether the session has been (manually or automatically) _paused_ or _completed_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7696a51d-074a-40d1-bf68-08adb2804823",
   "metadata": {},
   "outputs": [],
   "source": [
    "while api.sessions.find_one(project_uid, session_uid).status not in [\"paused\", \"completed\"]:\n",
    "    time.sleep(10)\n",
    "\n",
    "exposure_export_job_uid = api.sessions.create_and_enqueue_export_exposures(project_uid, session_uid)\n",
    "particle_export_job_uid = api.sessions.create_and_enqueue_export_particles(project_uid, session_uid)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}