Fix up documentation for specific LLMs (#1217)

* Update LOCAL_LLM_GUIDE.md

* Create WINDOWS_WSL_GUIDE.md

* update docs

* Update docs/Agents.md

* update windows docs

---------

Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>

README.md

@@ -117,7 +117,7 @@ After completing the MVP, the team will focus on research in various areas, incl
 * OpenDevin is still an alpha project. It is changing very quickly and is unstable. We are working on getting a stable release out in the coming weeks.
 * OpenDevin will issue many prompts to the LLM you configure. Most of these LLMs cost money--be sure to set spending limits and monitor usage.
 * OpenDevin runs `bash` commands within a Docker sandbox, so it should not affect your machine. But your workspace directory will be attached to that sandbox, and files in the directory may be modified or deleted.
-* Our default Agent is currently the MonologueAgent, which has limited capabilities, but is fairly stable. We're working on other Agent implementations, including [SWE Agent](https://swe-agent.com/). You can [read about our current set of agents here](./docs/documentation/Agents.md).
+* Our default Agent is currently the MonologueAgent, which has limited capabilities, but is fairly stable. We're working on other Agent implementations, including [SWE Agent](https://swe-agent.com/). You can [read about our current set of agents here](./docs/Agents.md).
 
 ## 🚀 Get Started
 The easiest way to run OpenDevin is inside a Docker container.
@@ -144,6 +144,8 @@ If you want to use the (unstable!) bleeding edge, you can use `ghcr.io/opendevin
 
 See [Development.md](Development.md) for instructions on running OpenDevin without Docker.
 
+If you're running on Windows and having trouble, check out our [guide for Windows users](./docs/guides/Windows.md)
+
 ## 🤖 LLM Backends
 OpenDevin can work with any LLM backend.
 For a full list of the LM providers and models available, please consult the
@@ -159,13 +161,20 @@ The following environment variables might be necessary for some LLMs:
 * `LLM_DEPLOYMENT_NAME`
 * `LLM_API_VERSION`
 
+We have a few guides for running OpenDevin with specific model providers:
+* [ollama](./docs/guides/LocalLLMs.md)
+* [Azure](./docs/guides/AzureLLMs.md)
+
+If you're using another provider, we encourage you to open a PR to share your setup!
+
 **Note on Alternative Models:**
-Some alternative models may prove more challenging to tame than others.
-Fear not, brave adventurer! We shall soon unveil LLM-specific documentation to guide you on your quest.
-And if you've already mastered the art of wielding a model other than OpenAI's GPT,
-we encourage you to [share your setup instructions with us](https://github.com/OpenDevin/OpenDevin/issues/417).
+The best models are GPT-4 and Claude 3. Current local and open source models are
+not nearly as powerful. When using an alternative model,
+you may see long wait times between messages,
+poor responses, or errors about malformed JSON. OpenDevin
+can only be as powerful as the models driving it--fortunately folks on our team
+are actively working on building better open source models!
 
-There is also [documentation for running with local models using ollama](./docs/documentation/LOCAL_LLM_GUIDE.md).
 
 ## ⭐️ Research Strategy
 

docs/documentation/Agents.md → docs/Agents.md

@@ -33,7 +33,7 @@ Short term memory is stored as a Monologue object and the model can condense it
 
 `_initialize`: Utilizes the `INITIAL_THOUGHTS` list to give the agent a context for its capabilities and how to navigate the `/workspace`
 
-`step`: Modifies the current state by adding the most rescent actions and observations, then prompts the model to think about its next action to take. 
+`step`: Modifies the current state by adding the most recent actions and observations, then prompts the model to think about its next action to take.
 
 `search_memory`: Uses `VectorIndexRetriever` to find related memories within the long term memory.
 
@@ -91,6 +91,6 @@ The Code Act Agent is a minimalist agent. The agent works by passing the model a
 ### Methods:
 `__init__`: Initializes an agent with `llm` and a list of messages `List[Mapping[str, str]]`
 
-`step`: First, gets messages from state and then compiles them into a list for context. Next, pass the context list with the prompt to get the next command to execute. Finally, Execute command if valid, else return `AgentEchoAction(INVALID_INPUT_MESSAGE)` 
+`step`: First, gets messages from state and then compiles them into a list for context. Next, pass the context list with the prompt to get the next command to execute. Finally, Execute command if valid, else return `AgentEchoAction(INVALID_INPUT_MESSAGE)`
 
-`search_memory`: Not yet implemented
+`search_memory`: Not yet implemented

docs/documentation/LOCAL_LLM_GUIDE.md

@@ -1,108 +0,0 @@
-# Local LLM Guide with Ollama server
-
-## 0. Install Ollama:
-run the following command in a conda env with CUDA etc.
-
-Linux:
-```
-curl -fsSL https://ollama.com/install.sh | sh
-```
-Windows or macOS:
-
-- Download from [here](https://ollama.com/download/)
-
-## 1. Install Models:
-Ollama model names can be found [here](https://ollama.com/library) (See example below)
-
-![alt text](images/ollama.png)
-
-Once you have found the model you want to use copy the command and run it in your conda env.
-
-Example of llama2 q4 quantized:
-```
-conda activate <env_name>
-ollama run llama2:13b-chat-q4_K_M
-```
-
-you can check which models you have downloaded like this:
-```
-~$ ollama list
-NAME                            ID              SIZE    MODIFIED
-llama2:latest                   78e26419b446    3.8 GB  6 weeks ago
-mistral:7b-instruct-v0.2-q4_K_M eb14864c7427    4.4 GB  2 weeks ago
-starcoder2:latest               f67ae0f64584    1.7 GB  19 hours ago
-```
-## 2. Run Ollama in CLI:
-This command starts up the ollama server that is on port `11434`
-
-This will show the requests in CLI
-```
-conda activate <env_name>
-ollama serve
-```
-or
-
-This will run with no output in the background
-```
-sudo systemctl start ollama
-```
-
-If you see something like this:
-```
-Error: listen tcp 127.0.0.1:11434: bind: address already in use
-```
-This is not an error it just means the server is already running
-
-To stop the server use:
-```
-sudo systemctl stop ollama
-```
-
-For more info go [here](https://github.com/ollama/ollama/blob/main/docs/faq.md)
-
-## 3. Start OpenDevin
-
-Use the instructions in [README.md](/README.md) to start OpenDevin using Docker.
-When running `docker run`, add the following environment variables using `-e`:
-
-```
-LLM_API_KEY="ollama"
-LLM_MODEL="ollama/<model_name>"
-LLM_EMBEDDING_MODEL="local"
-LLM_BASE_URL="http://localhost:<port_number>"
-WORKSPACE_DIR="./workspace"
-```
-Notes:
-- The API key should be set to `"ollama"`
-- The base url needs to be `localhost`
-- By default ollama port is `11434` unless you set it
-- `model_name` needs to be the entire model name
-    - Example: `LLM_MODEL="ollama/llama2:13b-chat-q4_K_M"`
-
-You should now be able to connect to `http://localhost:3001/` with your local model running!
-
-
-## Additional Notes for WSL2 Users:
-
-1. If you encounter the following error during setup: `Exception: Failed to create opendevin user in sandbox: b'useradd: UID 0 is not unique\n'`
-You can resolve it by running:
-    ```
-    export SANDBOX_USER_ID=1000
-    ```
-
-2. If you face issues running Poetry even after installing it during the build process, you may need to add its binary path to your environment:
-    ```
-    export PATH="$HOME/.local/bin:$PATH"
-    ```
-
-3. If you experiencing issues related to networking, such as `NoneType object has no attribute 'request'` when executing `make run`, you may need to configure your WSL2 networking settings. Follow these steps:
-   - Open or create the `.wslconfig` file located at `C:\Users\%username%\.wslconfig` on your Windows host machine.
-   - Add the following configuration to the `.wslconfig` file:
-        ```
-        [wsl2]
-        networkingMode=mirrored
-        localhostForwarding=true
-        ```
-   - Save the `.wslconfig` file.
-   - Restart WSL2 completely by exiting any running WSL2 instances and executing the command `wsl --shutdown` in your command prompt or terminal.
-   - After restarting WSL, attempt to execute `make run` again. The networking issue should be resolved.

docs/guides/LocalLLMs.md

@@ -0,0 +1,67 @@
+# Local LLM Guide with Ollama server
+
+## 0. Install and Start ollama:
+run the following command in a conda env with CUDA etc.
+
+Linux:
+```
+curl -fsSL https://ollama.com/install.sh | sh
+```
+Windows or macOS:
+
+- Download from [here](https://ollama.com/download/)
+
+Then run:
+```bash
+ollama serve
+```
+
+## 1. Install Models:
+Ollama model names can be found [here](https://ollama.com/library). For a small example, you can use
+the codellama:7b model. Bigger models will generally perform better.
+
+```
+ollama pull codellama:7b
+```
+
+you can check which models you have downloaded like this:
+```
+~$ ollama list
+NAME                            ID              SIZE    MODIFIED
+llama2:latest                   78e26419b446    3.8 GB  6 weeks ago
+mistral:7b-instruct-v0.2-q4_K_M eb14864c7427    4.4 GB  2 weeks ago
+starcoder2:latest               f67ae0f64584    1.7 GB  19 hours ago
+```
+
+## 3. Start OpenDevin
+
+Use the instructions in [README.md](/README.md) to start OpenDevin using Docker.
+When running `docker run`, add the following environment variables using `-e`:
+
+```bash
+LLM_API_KEY="ollama"
+LLM_BASE_URL="http://localhost:11434"
+```
+
+For example:
+```bash
+# The directory you want OpenDevin to modify. MUST be an absolute path!
+export WORKSPACE_DIR=$(pwd)/workspace
+
+docker run \
+    -e LLM_API_KEY="ollama" \
+    -e LLM_BASE_URL="http://localhost:11434"
+    -e WORKSPACE_MOUNT_PATH=$WORKSPACE_DIR \
+    -v $WORKSPACE_DIR:/opt/workspace_base \
+    -v /var/run/docker.sock:/var/run/docker.sock \
+    -p 3000:3000 \
+    ghcr.io/opendevin/opendevin:main
+```
+
+You should now be able to connect to `http://localhost:3001/`
+
+## 4. Select your Model
+In the OpenDevin UI, click on the Settings wheel in the bottom-left corner.
+Then in the `Model` input, enter `codellama:7b`, or the name of the model you pulled earlier, and click Save.
+
+And now you're ready to go!

docs/guides/Windows.md

@@ -0,0 +1,31 @@
+# Notes for Windows and WSL Users
+OpenDevin only supports Windows via [WSL](https://learn.microsoft.com/en-us/windows/wsl/install).
+Please be sure to run all commands inside your WSL terminal.
+
+## Troubleshooting
+
+### Failed to create opendevin user
+If you encounter the following error during setup: `Exception: Failed to create opendevin user in sandbox: b'useradd: UID 0 is not unique\n'`
+You can resolve it by running:
+    ```
+    export SANDBOX_USER_ID=1000
+    ```
+
+### Poetry Installation
+If you face issues running Poetry even after installing it during the build process, you may need to add its binary path to your environment:
+    ```
+    export PATH="$HOME/.local/bin:$PATH"
+    ```
+
+### NoneType object has no attribute 'request'
+If you experiencing issues related to networking, such as `NoneType object has no attribute 'request'` when executing `make run`, you may need to configure your WSL2 networking settings. Follow these steps:
+- Open or create the `.wslconfig` file located at `C:\Users\%username%\.wslconfig` on your Windows host machine.
+- Add the following configuration to the `.wslconfig` file:
+```
+[wsl2]
+networkingMode=mirrored
+localhostForwarding=true
+```
+- Save the `.wslconfig` file.
+- Restart WSL2 completely by exiting any running WSL2 instances and executing the command `wsl --shutdown` in your command prompt or terminal.
+- After restarting WSL, attempt to execute `make run` again. The networking issue should be resolved.