Implement Jupyter Frontend (#1363)

* initialize plugin definition

* initialize plugin definition

* simplify mixin

* further improve plugin mixin

* add cache dir for pip

* support clean up cache

* add script for setup jupyter and execution server

* integrate JupyterRequirement to ssh_box

* source bashrc at the end of plugin load

* add execute_cli that accept code via stdin

* make JUPYTER_EXEC_SERVER_PORT configurable via env var

* increase background cmd sleep time

* Update opendevin/sandbox/plugins/mixin.py

Co-authored-by: Robert Brennan <accounts@rbren.io>

* add mixin to base class

* make jupyter requirement a dataclass

* source plugins only when >0 requirements

* add `sandbox_plugins` for each agent & have controller take care of it

* update build.sh to make logs available in /opendevin/logs

* switch to use config for lib and cache dir

* Add SANDBOX_WORKSPACE_DIR into config

* Add SANDBOX_WORKSPACE_DIR into config

* fix occurence of /workspace

* fix permission issue with /workspace

* use python to implement execute_cli to avoid stdin escape issue

* add IPythonRunCellAction and get it working

* wait until jupyter is avaialble

* support plugin via copying instead of mounting

* add agent talk action

* support follow-up user language feedback

* add __str__ for action to be printed better

* only print PLAN at the beginning

* wip: update codeact agent

* get rid the initial messate

* update codeact agent to handle null action;
add thought to bash

* dispatch thought for RUN action as well

* fix weird behavior of pxssh where the output would not flush correctly

* make ssh box can handle exit_code properly as well

* add initial version of swe-agent plugin;

* rename swe cursors

* split setup script into two and create two requirements

* print SWE-agent command documentation

* update swe-agent to default to no custom docs

* add initial version of swe-agent plugin;

* rename swe cursors

* split setup script into two and create two requirements

* print SWE-agent command documentation

* update swe-agent to default to no custom docs

* update dockerfile with dependency from swe-agent

* make env setup a separate script for .bashrc source

* add wip prompt

* fix mount_dir for ssh_box

* update prompt

* fix mount_dir for ssh_box

* default to use host network

* default to use host network

* move prompt to a separate file

* fix swe-tool plugins;
add missing _split_string

* remove hostname from sshbox

* update the prompt with edit functionality

* fix swe-tool plugins;
add missing _split_string

* add awaiting into status bar

* fix the bug of additional send event

* remove some print action

* move logic to config.py

* remove debugging comments

* make host network as default

* make WORKSPACE_MOUNT_PATH as abspath

* implement execute_cli via file cp

* Revert "implement execute_cli via file cp"

This reverts commit 06f0155bc17d1f99097e71b83b2143f6e8092654.

* add codeact dependencies to default container

* add IPythonRunCellObservation

* add back cache dir and default to /tmp

* make USE_HOST_NETWORK a bool

* revert use host network to false

* add temporarily fix for IPython RUN action

* preliminary implementation of CodeActAgent's jupyter

* update node module

* update prompt

* revert USE_HOST_NETWORK to true since it is not affecting anything

* attempt to fix lint

* remove newline

* update prompt

* Refactor browser style. (#1358)

* delete useless assets and css class.
* add waiting for page loaded (networkidle with 3s timeout)

* Add integration test framework with mock llm (#1301)

* Add integration test framework with mock llm

* Fix MonologueAgent and PlannerAgent tests

* Remove adhoc logging

* Use existing logs

* Fix SWEAgent and PlannerAgent

* Check-in test log files

* conftest: look up under test name folder only

* Add docstring to conftest

* Finish dev doc

* Avoid non-determinism

* Remove dependency on llm embedding model

* Init embedding model only for MonologueAgent

* Add adhoc fix for sandbox discrepancy

* Test ssh and exec sandboxes

* CI: fix missing sandbox type

* conftest: Remove hack

* Reword comment for TODO

* Revert "refactor(frontend): Terminal (#1315)" (#1360)

This reverts commit 27246aca7e0f3d399740db466f31026c891a5374.

* revert USE_HOST_NETWORK to true since it is not affecting anything

* attempt to fix lint

* handle IsADirectory errors (#1365)

* update to 0.4.0 (#1362)

Co-authored-by: Jim Su <jimsu@protonmail.com>

* feat(frontend): multiple design changes (#1370)

* fix/improve terminal hook (#1371)

* Revert "update node module"

This reverts commit 459b1031e722529ddc00ca475b88245bf52edeaa.

* support SyntaxHighlighter and markdown for jupyter visualization

* fix jupyter execution server

* make jupyter active

* improve the display of markdown and raw text

* get base64 image display for react

* add `thought` to most action class

* fix unit tests for current action abstraction

* support user exit

* update test cases with the latest action format (added 'thought')

* fix integration test for CodeActAGent by mocking stdin

* only mock stdin for tests with user_responses.log

* remove -exec integration test for CodeActAgent since it is not supported

* remove specific stop word

* fix comments

* improve clarity of prompt

* attempt to fix lint

* attempt to fix lint yet agiain

* fix py lint

* fix integration tests

* sandbox might failed in chown due to mounting, but it won't be fatal

* update debug instruction for sshbox

* fix typo

* get RUN_AS_DEVIN and network=host working with app sandbox

* get RUN_AS_DEVIN and network=host working with app sandbox

* attempt to fix the workspace base permission

* sandbox might failed in chown due to mounting, but it won't be fatal

* update sshbox instruction

* remove default user id since it will be passed in the instruction

* revert permission fix since it should be resolved by correct SANDBOX_USER_ID

* the permission issue can be fixed by simply provide correct env var

* remove log

* set sandbox user id to getuid by default

* move logging to initializer

* make the uid consistent across host, app container, and sandbox

* remove hostname as it causes sudo issue

* fix permission of entrypoint script

* make the uvicron app run as host user uid for jupyter plugin

* add warning message

* fix frontend lint

* update dev md for instruction of running unit tests

* add back unit tests

* revert back to the original sandbox implementation to fix testcases

* revert use host network

* get docker socket gid and usermod instead of chmod 777

* allow unit test workflow to find docker.sock

* make sandbox test working via patch

* fix arg parser that's broken for some reason

* try to fix app build disk space issue

* fix integration test

* Revert "fix arg parser that's broken for some reason"

This reverts commit 6cc89611337bb74555fd16b4be78681fb7e36573.

* update Development.md

* cleanup intergration tests & add exception for CodeAct+execbox

* fix config

* implement user_message action

* fix doc

* fix event dict error

* fix frontend lint

* revert accidentally changes to integration tests

* revert accidentally changes to integration tests

---------

Co-authored-by: Robert Brennan <accounts@rbren.io>
Co-authored-by: Leo <ifuryst@gmail.com>
Co-authored-by: Boxuan Li <liboxuan@connect.hku.hk>
Co-authored-by: Jim Su <jimsu@protonmail.com>
Co-authored-by: Alex Bäuerle <alex@a13x.io>
Co-authored-by: sp.wack <83104063+amanape@users.noreply.github.com>
Co-authored-by: Robert Brennan <contact@rbren.io>

agenthub/codeact_agent/codeact_agent.py

@@ -122,6 +122,12 @@ class CodeActAgent(Agent):
                     self.messages.append({'role': 'user', 'content': content})
                 elif isinstance(obs, IPythonRunCellObservation):
                     content = 'OBSERVATION:\n' + obs.content
+                    # replace base64 images with a placeholder
+                    splited = content.split('\n')
+                    for i, line in enumerate(splited):
+                        if '![image](data:image/png;base64,' in line:
+                            splited[i] = '![image](data:image/png;base64, ...) already displayed to user'
+                    content = '\n'.join(splited)
                     self.messages.append({'role': 'user', 'content': content})
                 else:
                     raise NotImplementedError(

frontend/package-lock.json

@@ -26,6 +26,7 @@
         "react": "^18.2.0",
         "react-accessible-treeview": "^2.8.3",
         "react-dom": "^18.2.0",
+        "react-highlight": "^0.15.0",
         "react-hot-toast": "^2.4.1",
         "react-i18next": "^14.1.0",
         "react-icons": "^5.0.1",
@@ -44,6 +45,7 @@
         "@types/node": "^18.0.0 ",
         "@types/react": "^18.2.66",
         "@types/react-dom": "^18.2.22",
+        "@types/react-highlight": "^0.12.8",
         "@types/react-syntax-highlighter": "^15.5.11",
         "@typescript-eslint/eslint-plugin": "^7.4.0",
         "@typescript-eslint/parser": "^7.0.0",
@@ -4878,6 +4880,15 @@
         "@types/react": "*"
       }
     },
+    "node_modules/@types/react-highlight": {
+      "version": "0.12.8",
+      "resolved": "https://registry.npmjs.org/@types/react-highlight/-/react-highlight-0.12.8.tgz",
+      "integrity": "sha512-V7O7zwXUw8WSPd//YUO8sz489J/EeobJljASGhP0rClrvq+1Y1qWEpToGu+Pp7YuChxhAXSgkLkrOYpZX5A62g==",
+      "dev": true,
+      "dependencies": {
+        "@types/react": "*"
+      }
+    },
     "node_modules/@types/react-syntax-highlighter": {
       "version": "15.5.11",
       "resolved": "https://registry.npmjs.org/@types/react-syntax-highlighter/-/react-syntax-highlighter-15.5.11.tgz",
@@ -12799,6 +12810,14 @@
         "react": "^18.2.0"
       }
     },
+    "node_modules/react-highlight": {
+      "version": "0.15.0",
+      "resolved": "https://registry.npmjs.org/react-highlight/-/react-highlight-0.15.0.tgz",
+      "integrity": "sha512-5uV/b/N4Z421GSVVe05fz+OfTsJtFzx/fJBdafZyw4LS70XjIZwgEx3Lrkfc01W/RzZ2Dtfb0DApoaJFAIKBtA==",
+      "dependencies": {
+        "highlight.js": "^10.5.0"
+      }
+    },
     "node_modules/react-hot-toast": {
       "version": "2.4.1",
       "resolved": "https://registry.npmjs.org/react-hot-toast/-/react-hot-toast-2.4.1.tgz",

frontend/package.json

@@ -25,6 +25,7 @@
     "react": "^18.2.0",
     "react-accessible-treeview": "^2.8.3",
     "react-dom": "^18.2.0",
+    "react-highlight": "^0.15.0",
     "react-hot-toast": "^2.4.1",
     "react-i18next": "^14.1.0",
     "react-icons": "^5.0.1",
@@ -64,6 +65,7 @@
     "@types/node": "^18.0.0 ",
     "@types/react": "^18.2.66",
     "@types/react-dom": "^18.2.22",
+    "@types/react-highlight": "^0.12.8",
     "@types/react-syntax-highlighter": "^15.5.11",
     "@typescript-eslint/eslint-plugin": "^7.4.0",
     "@typescript-eslint/parser": "^7.0.0",

frontend/src/components/Jupyter.tsx

@@ -0,0 +1,77 @@
+import React from "react";
+import { useSelector } from "react-redux";
+import SyntaxHighlighter from "react-syntax-highlighter";
+import Markdown from "react-markdown";
+import { atomOneDark } from "react-syntax-highlighter/dist/esm/styles/hljs";
+import { RootState } from "#/store";
+import { Cell } from "#/state/jupyterSlice";
+
+interface IJupyterCell {
+  cell: Cell;
+}
+
+function JupyterCell({ cell }: IJupyterCell): JSX.Element {
+  const code = cell.content;
+
+  if (cell.type === "input") {
+    return (
+      <div className="rounded-lg bg-gray-800 dark:bg-gray-900 p-2 text-xs">
+        <div className="mb-1 text-gray-400">EXECUTE</div>
+        <pre
+          className="scrollbar-custom scrollbar-thumb-gray-500 hover:scrollbar-thumb-gray-400 dark:scrollbar-thumb-white/10 dark:hover:scrollbar-thumb-white/20 overflow-auto px-5"
+          style={{ padding: 0, marginBottom: 0, fontSize: "0.75rem" }}
+        >
+          <SyntaxHighlighter language="python" style={atomOneDark}>
+            {code}
+          </SyntaxHighlighter>
+        </pre>
+      </div>
+    );
+  }
+  return (
+    <div className="rounded-lg bg-gray-800 dark:bg-gray-900 p-2 text-xs">
+      <div className="mb-1 text-gray-400">STDOUT/STDERR</div>
+      <pre
+        className="scrollbar-custom scrollbar-thumb-gray-500 hover:scrollbar-thumb-gray-400 dark:scrollbar-thumb-white/10 dark:hover:scrollbar-thumb-white/20 overflow-auto px-5 max-h-[60vh] bg-gray-800"
+        style={{ padding: 0, marginBottom: 0, fontSize: "0.75rem" }}
+      >
+        {/* split code by newline and render each line as a plaintext, except it starts with `![image]` so we render it as markdown */}
+        {code.split("\n").map((line, index) => {
+          if (line.startsWith("![image](data:image/png;base64,")) {
+            // add new line before and after the image
+            return (
+              <div key={index}>
+                <Markdown urlTransform={(value: string) => value}>
+                  {line}
+                </Markdown>
+                <br />
+              </div>
+            );
+          }
+          return (
+            <div key={index}>
+              <SyntaxHighlighter language="plaintext" style={atomOneDark}>
+                {line}
+              </SyntaxHighlighter>
+              <br />
+            </div>
+          );
+        })}
+      </pre>
+    </div>
+  );
+}
+
+function Jupyter(): JSX.Element {
+  const { cells } = useSelector((state: RootState) => state.jupyter);
+
+  return (
+    <div className="flex-1 overflow-y-auto flex flex-col">
+      {cells.map((cell, index) => (
+        <JupyterCell key={index} cell={cell} />
+      ))}
+    </div>
+  );
+}
+
+export default Jupyter;

frontend/src/components/Workspace.tsx

@@ -12,6 +12,7 @@ import { AllTabs, TabOption, TabType } from "#/types/TabOption";
 import Browser from "./Browser";
 import CodeEditor from "./CodeEditor";
 import Planner from "./Planner";
+import Jupyter from "./Jupyter";
 
 function Workspace() {
   const { t } = useTranslation();
@@ -20,12 +21,13 @@ function Workspace() {
   const screenshotSrc = useSelector(
     (state: RootState) => state.browser.screenshotSrc,
   );
-
+  const jupyterCells = useSelector((state: RootState) => state.jupyter.cells);
   const [activeTab, setActiveTab] = useState<TabType>(TabOption.CODE);
   const [changes, setChanges] = useState<Record<TabType, boolean>>({
     [TabOption.PLANNER]: false,
     [TabOption.CODE]: false,
     [TabOption.BROWSER]: false,
+    [TabOption.JUPYTER]: false,
   });
 
   const tabData = useMemo(
@@ -45,6 +47,11 @@ function Workspace() {
         icon: <IoIosGlobe size={18} />,
         component: <Browser key="browser" />,
       },
+      [TabOption.JUPYTER]: {
+        name: t(I18nKey.WORKSPACE$JUPYTER_TAB_LABEL),
+        icon: <VscCode size={18} />,
+        component: <Jupyter key="jupyter" />,
+      },
     }),
     [t],
   );
@@ -73,6 +80,14 @@ function Workspace() {
     // eslint-disable-next-line react-hooks/exhaustive-deps
   }, [screenshotSrc]);
 
+  useEffect(() => {
+    if (activeTab !== TabOption.JUPYTER && jupyterCells.length > 0) {
+      // FIXME: This is a temporary solution to show the jupyter tab when the first cell is added
+      // Only need to show the tab only when a cell is added
+      setChanges((prev) => ({ ...prev, [TabOption.JUPYTER]: true }));
+    }
+  }, [jupyterCells]);
+
   return (
     <div className="flex flex-col min-h-0 grow">
       <div

frontend/src/i18n/translation.json

@@ -39,6 +39,19 @@
     "pt": "Planejador",
     "es": "Planificador"
   },
+  "WORKSPACE$JUPYTER_TAB_LABEL": {
+    "en": "Jupyter IPython",
+    "zh-CN": "Jupyter IPython",
+    "de": "Jupyter IPython",
+    "ko-KR": "Jupyter IPython",
+    "no": "Jupyter IPython",
+    "zh-TW": "Jupyter IPython",
+    "ar": "Jupyter IPython",
+    "fr": "Jupyter IPython",
+    "it": "Jupyter IPython",
+    "pt": "Jupyter IPython",
+    "es": "Jupyter IPython"
+  },
   "WORKSPACE$CODE_EDITOR_TAB_LABEL": {
     "en": "Code Editor",
     "zh-CN": "代码编辑器",

frontend/src/services/actions.ts

@@ -3,6 +3,7 @@ import { setScreenshotSrc, setUrl } from "#/state/browserSlice";
 import { appendAssistantMessage } from "#/state/chatSlice";
 import { setCode, updatePath } from "#/state/codeSlice";
 import { appendInput } from "#/state/commandSlice";
+import { appendJupyterInput } from "#/state/jupyterSlice";
 import { setPlan } from "#/state/planSlice";
 import { setInitialized } from "#/state/taskSlice";
 import store from "#/store";
@@ -45,7 +46,7 @@ const messageActions = {
     if (message.args.thought) {
       store.dispatch(appendAssistantMessage(message.args.thought));
     }
-    store.dispatch(appendInput(message.args.code));
+    store.dispatch(appendJupyterInput(message.args.code));
   },
   [ActionType.ADD_TASK]: () => {
     getPlan().then((fetchedPlan) => store.dispatch(setPlan(fetchedPlan)));

frontend/src/services/observations.ts

@@ -3,6 +3,7 @@ import { setUrl, setScreenshotSrc } from "#/state/browserSlice";
 import store from "#/store";
 import { ObservationMessage } from "#/types/Message";
 import { appendOutput } from "#/state/commandSlice";
+import { appendJupyterOutput } from "#/state/jupyterSlice";
 import ObservationType from "#/types/ObservationType";
 
 export function handleObservationMessage(message: ObservationMessage) {
@@ -12,7 +13,7 @@ export function handleObservationMessage(message: ObservationMessage) {
       break;
     case ObservationType.RUN_IPYTHON:
       // FIXME: render this as markdown
-      store.dispatch(appendOutput(message.content));
+      store.dispatch(appendJupyterOutput(message.content));
       break;
     case ObservationType.BROWSE:
       if (message.extras?.screenshot) {

frontend/src/state/jupyterSlice.ts

@@ -0,0 +1,27 @@
+import { createSlice } from "@reduxjs/toolkit";
+
+export type Cell = {
+  content: string;
+  type: "input" | "output";
+};
+
+const initialCells: Cell[] = [];
+
+export const cellSlice = createSlice({
+  name: "cell",
+  initialState: {
+    cells: initialCells,
+  },
+  reducers: {
+    appendJupyterInput: (state, action) => {
+      state.cells.push({ content: action.payload, type: "input" });
+    },
+    appendJupyterOutput: (state, action) => {
+      state.cells.push({ content: action.payload, type: "output" });
+    },
+  },
+});
+
+export const { appendJupyterInput, appendJupyterOutput } = cellSlice.actions;
+
+export default cellSlice.reducer;

frontend/src/store.ts

@@ -7,6 +7,7 @@ import commandReducer from "./state/commandSlice";
 import errorsReducer from "./state/errorsSlice";
 import planReducer from "./state/planSlice";
 import taskReducer from "./state/taskSlice";
+import jupyterReducer from "./state/jupyterSlice";
 
 export const rootReducer = combineReducers({
   browser: browserReducer,
@@ -17,6 +18,7 @@ export const rootReducer = combineReducers({
   errors: errorsReducer,
   plan: planReducer,
   agent: agentReducer,
+  jupyter: jupyterReducer,
 });
 
 const store = configureStore({

frontend/src/types/TabOption.tsx

@@ -2,10 +2,20 @@ enum TabOption {
   PLANNER = "planner",
   CODE = "code",
   BROWSER = "browser",
+  JUPYTER = "jupyter",
 }
 
-type TabType = TabOption.PLANNER | TabOption.CODE | TabOption.BROWSER;
+type TabType =
+  | TabOption.PLANNER
+  | TabOption.CODE
+  | TabOption.BROWSER
+  | TabOption.JUPYTER;
 
-const AllTabs = [TabOption.CODE, TabOption.BROWSER, TabOption.PLANNER];
+const AllTabs = [
+  TabOption.CODE,
+  TabOption.BROWSER,
+  TabOption.PLANNER,
+  TabOption.JUPYTER,
+];
 
 export { AllTabs, TabOption, type TabType };

opendevin/sandbox/plugins/jupyter/execute_server

@@ -187,8 +187,7 @@ class JupyterKernel:
                     outputs.append(msg['content']['data']['text/plain'])
                     if 'image/png' in msg['content']['data']:
                         # use markdone to display image (in case of large image)
-                        # outputs.append(f"\n<img src=\'data:image/png;base64,{msg['content']['data']['image/png']}\'/>\n")
-                        outputs.append(f"![image](data:image/png;base64,{msg['content']['data']['image/png']})")
+                        outputs.append(f"\n![image](data:image/png;base64,{msg['content']['data']['image/png']})\n")
 
                 elif msg_type == 'execute_reply':
                     execution_done = True