This post is very off-topic for me, but I just spent a lot of time trying to figure this out and figured I would make a post here in case it could help someone out in the future. Dig in!
Render.com is a nice way to quickly deploy web applications without a lot of effort / hassle. You can link your app’s repository on github.com to Render, and when you push changes to the main repository branch, it will execute a build step and deploy your app to a public domain with HTTPS, so I really like it for little projects I’m working on.
Right now I am working on an API endpoint for my portfolio optimizer. It is a simple Python FastAPI endpoint that takes in a few parameters from the optimizer frontend and runs a proprietary optimizer algorithm to return optimized portfolio weights. The content of the project isn’t important, the key thing is that I have an API that I want to build and deploy to Render and there is a piece that has some private business logic that I would like to maintain as a completely separate Python package as a dependency.
I have been using Poetry lately to manage package dependencies for new Python projects. I like it, it reminds me of Rust’s Cargo system which is one of the main things I like about the Rust language. Poetry defines a pyproject.toml file for dependencies and a poetry.lock file for version pinning. So in my API project, I had a directory structure like this:
API/
__init__.py
main.py
tests/
__init__.py
.gitignore
poetry.lock
pyproject.toml
README.mdAnd a pyproject.toml like this:
[tool.poetry]
name = "my-api"
version = "0.1.0"
description = ""
authors = ["Justin Luther <justinluther@gmail.com>"]
readme = "README.md"
packages = [{include = "my_api"}]
[tool.poetry.dependencies]
python = ">3.10, <3.12"
fastapi = "^0.95.2"
uvicorn = {extras = ["standard"], version = "^0.22.0"}
my-private-dependency = {git = "ssh://git@github.com/myorganization/myprivaterepo.git"}
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
This works locally, because both the main repo and private dependency can be accessed from github, authenticated by the same SSH key stored in my local machine at ~/.ssh/. However, I noticed that when the project was pushed to render.com, the build step was failing and the logs showed that the render container was not able to clone the private dependency repo, returning an error “Could not read from remote repository. Please make sure you have the correct access rights.”
I did some googling and found that some other people had the same issue, including with other types of projects like NPM, and had found various workarounds. But many of the workarounds I found involved basically pasting the private dependency code into the API repo, which I didn’t want to do. I wanted to keep the module separate so that I could update and maintain the private dependency independently of the API.
So basically render is correctly authenticating for the main repo (also private on github) but not for the dependency repo. This was unexpected because they are hosted in the same github account but apparently the render container is not authenticating to github correctly through “poetry install”.
The first thing I tried was to add a github “deploy key” to the dependency repo. Deploy keys allow fine-grained access (read only by default) to individual github repos. So I went to render.com, loaded the shell prompt, and ran
$ ssh-keygento generate a key pair in ~/.ssh/ for my render container. I copied the public key and pasted it into the “add deploy key” section of github’s page for my private dependency. I’m hoping that now when “poetry install” runs when my container builds, it will find my private deploy key and correctly authenticate with github to install the private dependency.
Nope! I get the same error message and it appears that the key that I generated is not in the right place for Poetry or Git to find it when “poetry install” runs in the build step. I know that running “ssh-keygen” from render’s shell prompt put the key in a directory called ~/.ssh, but my hunch is that “~” in my render container is not the regular home directory that Git and Poetry are looking for, and that I probably can’t put a key where Git and Poetry are going to look for it. I confirm this by outputting some debug echo statements in the render.com build step and find out that home, in this case, is /opt/render.
So sticking some keys in my render container directory somewhere is probably not going to work. My next thought was to store a private key in a render.com secret file, and then see if I can pass that file contents directly into a “git clone” statement, or “poetry install” statement. But that seemed dangerous and I vaguely worried that this approach could leak the private key in server logs, etc. And it turns out that neither git nor poetry has an option to set a manual authentication key through the shell command. I think.
After some more googling I stumbled upon Git submodules, which is Git’s own way of dealing with separately maintained dependencies apart from Poetry. You can run something like:
git submodule add git@github.com:username/reponame.gitand Git will clone a separate module into the parent directory structure, and also add a .gitmodules file to the parent root directory that looks like:
[submodule “name-of-submodule”]
path = submodule
url = git@github.com:username/submodule.git
And now my directory structure looks like:
API/
__init__.py
main.py
private-dependency/
private_dependency/
__init__.py
some_module.py
other_module.py
tests/
.gitignore
poetry.lock
pyproject.toml
README.md
tests/
__init__.py
.gitignore
poetry.lock
pyproject.toml
README.mdThis is exactly what I want, but I don’t want to manage dependencies separately with Git modules for the private dependency and Poetry for everything else. Ideally I’d like Poetry to manage everything and keep version requirements tidy.
So I dug more into the Poetry docs and found that Poetry does support Git submodules by supporting local path dependencies. So I changed my pythonproject.toml file to point to a local python package path rather than a github repository:
[tool.poetry]
name = “my-api”
version = “0.1.0”
description = “”
authors = [“Justin Luther <justinluther@gmail.com>”]
readme = “README.md”
packages = [{include = “my_api”}]
[tool.poetry.dependencies]
python = “>3.10, <3.12”
fastapi = “^0.95.2”
uvicorn = {extras = [“standard”], version = “^0.22.0”}
mpt-optimizer = {path = “./my-dependency”, develop = true}
[build-system]
requires = [“poetry-core”]
build-backend = “poetry.core.masonry.api”
When render.com clones the source API repo, it sets the –recursive flag, so that all Git submodules get cloned into the correct directory structure inside the container. That gets the dependency source code into the container, and then Poetry can find it through its local path dependency. The dependency repo gets cloned OK because it is done in the same step as the API repo, so that whatever render.com is doing to authenticate for the API repo will also work for the dependency repo.
So OK. After a few hours of frustration, I now have a render.com Python FastAPI project with a nicely modularized separate, private library for my optimizer algorithm, that I can maintain separately from my API.
The one thing I’ve noticed about this approach is that .gitmodules interacts strangely with Poetry. Sometimes when I update the dependency package, the code in the parent module doesn’t pick up the changes. Even if the working directory contains the updated code, the parent module still executes the code from the old module. I think this has to do with Poetry’s environment cache.
This isn’t a hug deal, I have found it helps to just remove the Poetry environment, update the git submodule, and then re-run “poetry install”. I’m sure there’s a better way to deal with this issue but this seems to work fine.