{"id":875,"date":"2024-11-15T16:16:40","date_gmt":"2024-11-15T16:16:40","guid":{"rendered":"https:\/\/realstudy.net\/?p=875"},"modified":"2024-11-19T01:45:34","modified_gmt":"2024-11-19T01:45:34","slug":"machine-learning-dev-environment-jupyterlab","status":"publish","type":"post","link":"https:\/\/realstudy.net\/?p=875","title":{"rendered":"Machine Learning Dev Environment \u2013 JupyterLab"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"875\" class=\"elementor elementor-875\" data-elementor-post-type=\"post\">\n\t\t\t\t<div class=\"elementor-element elementor-element-b192c59 e-flex e-con-boxed e-con e-parent\" data-id=\"b192c59\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-e84a208 elementor-widget elementor-widget-text-editor\" data-id=\"e84a208\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>A stable and consistent development environment is essential for machine learning and deep learning research. Setting up a JupyterLab environment that integrates major deep learning frameworks like TensorFlow and PyTorch can significantly boost productivity. This blog provides a step-by-step guide to running JupyterLab with TensorFlow and PyTorch pre-installed using Docker-Compose. With this setup, you can start machine learning experiments effortlessly, without the hassle of complex configurations.<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-75f36a5 elementor-widget elementor-widget-text-editor\" data-id=\"75f36a5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3>Prerequisites:<\/h3><ul style=\"margin-left: 0em;\"><li><strong>Docker<\/strong> and <strong>Docker-Compose<\/strong> installed<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b49ef25 elementor-widget elementor-widget-text-editor\" data-id=\"b49ef25\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3>Contents:<\/h3><ol style=\"margin-left: 0em;\"><li><strong>Installing NVIDIA Drivers:<\/strong><br \/><ul style=\"margin-left: 0em;\"><li>Install the appropriate driver for your GPU using apt.<\/li><\/ul><\/li><li><strong>Setting Up with Docker Compose:<\/strong><br \/><ul style=\"margin-left: 0em;\"><li>Run JupyterLab with TensorFlow and PyTorch pre-installed on your local machine using Docker.<\/li><\/ul><\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-70e61a0 e-flex e-con-boxed e-con e-parent\" data-id=\"70e61a0\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-7a12cf3 elementor-widget elementor-widget-heading\" data-id=\"7a12cf3\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Installing NVIDIA Drivers<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0f46a80 elementor-widget elementor-widget-text-editor\" data-id=\"0f46a80\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>* Identify the required driver version for your GPU and install it. If you don&#8217;t have a GPU, you can skip this step. Without a GPU, TensorFlow and PyTorch in JupyterLab will run on the CPU.<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-74da9e0 elementor-widget elementor-widget-code-highlight\" data-id=\"74da9e0\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp># check driver list\nuser:~$ ubuntu-drivers devices\n\n# install driver\nuser:~$ sudo apt install -y nvidia-driver-550-server<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a5796d1 elementor-widget elementor-widget-text-editor\" data-id=\"a5796d1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>* Additionally, there&#8217;s a popular &#8220;Graphics Drivers&#8221; team PPA that maintains newer NVIDIA drivers compatible with the latest versions of Ubuntu. This PPA is unofficial (not maintained by the Ubuntu team or NVIDIA) but is well-known. By adding this PPA, you can install more recent drivers. However, it&#8217;s not officially supported by Ubuntu, so it&#8217;s advisable not to use it on production machines.<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9f2662b elementor-widget elementor-widget-code-highlight\" data-id=\"9f2662b\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp># Add ppa\nuser:~$ sudo add-apt-repository ppa:graphics-drivers\/ppa\n\n# Update list\nuser:~$ sudo apt update<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-f32f7da e-flex e-con-boxed e-con e-parent\" data-id=\"f32f7da\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-9ec65eb elementor-widget elementor-widget-heading\" data-id=\"9ec65eb\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">* Resolving TensorFlow Warning - NUMA Error Encountered<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0ebca2c elementor-widget elementor-widget-text-editor\" data-id=\"0ebca2c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>* Install additional required libraries. If they are already installed, no installation is necessary.<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bb4ca65 elementor-widget elementor-widget-code-highlight\" data-id=\"bb4ca65\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp># Check if lspci is installed.\nuser:~$ which lspci\n\n# Install pciutils (optional)\nuser:~$ sudo apt install pciutils<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-be8efe1 elementor-widget elementor-widget-text-editor\" data-id=\"be8efe1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>* &#8220;NUMA node read from SysFS had negative value(-1)&#8221; \uc624\ub958 \ubc1c\uc0dd\ud560 \uacbd\uc6b0 \uc544\ub798\uc640 \uac19\uc774 NVIDIA GPU\uc758 NUMA \ub178\ub4dc \ud30c\uc77c \uacbd\ub85c \uc124\uc815\uc744 \uc704\ud55c PCI Device \ubc88\ud638 \ud655\uc778.\u00a0<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-434b548 elementor-widget elementor-widget-code-highlight\" data-id=\"434b548\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp># Check nvidia device\nuser:~$ lspci | grep -i nvidia\n\n01:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1)\n01:00.1 Audio device: NVIDIA Corporation Device 22ba (rev a1)<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-92ed4ad elementor-widget elementor-widget-text-editor\" data-id=\"92ed4ad\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>* In the above example, we confirmed that the NUMA node file path for the NVIDIA GPU is &#8220;01:00.0&#8221;. The error occurs because the NUMA node is set to &#8220;-1&#8221;, indicating that it is not assigned to a specific NUMA node. To resolve this, we set it to 0. Note that this setting will need to be reapplied after a reboot.<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c442b59 elementor-widget elementor-widget-code-highlight\" data-id=\"c442b59\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp># Set NUMA (\/sys\/bus\/pci\/devices\/0000:{PCI Device}\/numa_node)\nuser:~$ echo 0 | sudo tee -a \/sys\/bus\/pci\/devices\/0000\\:01\\:00.0\/numa_node<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7c9b661 elementor-widget elementor-widget-text-editor\" data-id=\"7c9b661\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>* To ensure the setting persists after a reboot, register it in the crontab to run once at boot time.<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6a1d85a elementor-widget elementor-widget-code-highlight\" data-id=\"6a1d85a\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp># Add this command to the end of the crontab\nuser:~$ suco crontab -e\n@reboot (echo 0 | sudo tee -a \/sys\/bus\/pci\/devices\/0000\\:01\\:00.0\/numa_node)<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-7f42a81 e-flex e-con-boxed e-con e-parent\" data-id=\"7f42a81\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-245b7b9 elementor-widget elementor-widget-heading\" data-id=\"245b7b9\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">* Setting Up Locally with Docker Compose<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-88f3d1a elementor-widget elementor-widget-text-editor\" data-id=\"88f3d1a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div>\n* In the example, the local path for storing Jupyter notebook files is set to \/data\/notebook, and the path for storing large datasets for training is set to \/data\/notebook_data.\n<\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-574a3bf elementor-widget elementor-widget-code-highlight\" data-id=\"574a3bf\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp>user:~$ sudo mkdir -p \/data\/notebook\nuser:~$ sudo mkdir -p \/data\/notebook_data<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-80770cb elementor-widget elementor-widget-text-editor\" data-id=\"80770cb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div><b>docker-compose.yml<\/b><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f4dbce4 elementor-widget elementor-widget-code-highlight\" data-id=\"f4dbce4\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>services:\n# define jupyterlab for machine learning\n  gupyterlab-tf-torch:\n    image: nockchun\/gupyterlab-tf-torch:2.17-2.5\n    restart: unless-stopped\n    container_name: gupyterlab\n    volumes:\n      - \/data\/notebook:\/notebook\n      - \/data\/notebook_data:\/data\n    ports:\n      - \"8888:8888\" # jupyterlab port\n      - \"6006:6006\" # tensorboard port\n    command:\n      jupyter lab --notebook-dir=\/notebook --no-browser --ip=0.0.0.0 --allow-root --NotebookApp.token=\"rsnet\"<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-84985ac elementor-widget elementor-widget-text-editor\" data-id=\"84985ac\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div><strong>* Commands to Start and Stop the Container:<\/strong><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c5f9ada elementor-widget elementor-widget-code-highlight\" data-id=\"c5f9ada\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-tomorrow copy-to-clipboard word-wrap\">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp># start container\nuser:~$ docker compose up gupyterlab-tf-torch -d\n\n# stop container\nuser:~$ docker container stop <container hash value>\n\n# remove container\nuser:~$ docker container rm <container hash value>\n<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>A stable and consistent development environment is essential for machine learning and deep learning research. Setting up a JupyterLab environment&hellip;<\/p>\n","protected":false},"author":1,"featured_media":593,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[15,21],"tags":[],"class_list":["post-875","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai_en","category-computer_en"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/realstudy.net\/wp-content\/uploads\/2024\/11\/jupyterlab_gpu.jpg","_links":{"self":[{"href":"https:\/\/realstudy.net\/index.php?rest_route=\/wp\/v2\/posts\/875","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/realstudy.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/realstudy.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/realstudy.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/realstudy.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=875"}],"version-history":[{"count":10,"href":"https:\/\/realstudy.net\/index.php?rest_route=\/wp\/v2\/posts\/875\/revisions"}],"predecessor-version":[{"id":932,"href":"https:\/\/realstudy.net\/index.php?rest_route=\/wp\/v2\/posts\/875\/revisions\/932"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/realstudy.net\/index.php?rest_route=\/wp\/v2\/media\/593"}],"wp:attachment":[{"href":"https:\/\/realstudy.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=875"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/realstudy.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=875"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/realstudy.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=875"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}