Local lean backtest on Mac fails

Hi Jesse,

This sounds like some sort of permission issue. Can you verify Docker has access to /var/folders by opening Docker and going to Settings > Resources > File sharing? It should look like this.

Hi Jasper,

I agree, that was the first thing I tried too (I didn't have that directory in the list before this).  I even rebooted the computer when it didn't work, and it still has the same result.

I also tried --verbose, and it prints the error stacktrace but can't see the docker command it's trying so not sure what else could be going wrong.  I checked the settings.json to make sure it's not corrupted in there or something and it shows:

"filesharingDirectories" : [
"\/Users\/jpangburn\/Documents\/amazon_aws\/aws_sam_projects",
"\/Users\/jpangburn\/Documents\/dockervolumes",
"\/tmp",
"\/var\/folders"
],

So it appears like that setting is correct.  But no idea what else it could be.  Thanks!

Hi Jesse,

That's weird, I'm not able to reproduce this error and don't really know what's going wrong here. Could you try giving Docker full disk access in macOS settings > Security & Privacy > Privacy? Looking at a similar issue on GitHub that may be a potential fix.

Hi Jasper,

Ah great idea!  I had looked at that similar issue as well, but didn't read that last comment.  Anyway, I tried it and still didn't work, even with rebooting the computer.  Still thinking it's a permission problem, I edited the temp_manager.py to make the temp directory under /tmp

But it had the same problem: Error: 400 Client Error for http+docker://localhost/v1.41/containers/create: Bad Request ("invalid mount config for type "bind": bind source path does not exist: /tmp/tmpdcjxmfgc/config.json")

Pretty bizarre as /tmp has been allowed under Docker forever and every user should have access to it.  For kicks I made a ~/tmp directory under my user account, and tried that.

This seems to have gotten past that error and found a new one: 

Error: 500 Server Error for http+docker://localhost/v1.41/containers/ab48c697a672f5a8afa9da29f9e0f119054cf00ee52c67f1c51667516bc97a20/start: Internal Server Error ("b'Mounts denied: \nThe path

/Users/jpangburn/Documents/vsc_workspaces/quantconnect_test/data is not shared from the host and is not known to Docker.\nYou can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.\nSee

https://docs.docker.com/docker-for-mac for more info.'")

It appears it's trying to bind a directory underneath of my project dir "/Users/jpangburn/Documents/vsc_workspaces/quantconnect_test".  I guess it makes sense since a backtest would need to access that data subdirectory.  But the setup guide said nothing about making this directory accessible to Docker's file sharing.  I guess in your Docker file sharing screenshot, you have /Users, /Volumes, and /private so that's why you didn't need to explicitly add this one.

I tried adding those 3 directories to my Docker file sharing, then it ran the backtest just fine.  I tried it again after removing Full Disk Access permission, and it still ran fine.  So seems like one or more of those shares is critical.  Unfortunately it also gets rid of the security you get by only allowing a small subset of directories on your machine to be accessed by Docker shared volumes, which is why they have that file sharing config in the first place.

Digging further, it seems you need /private/var/folders rather than /var/folders because /var is really mapped to /private/var.  So in summary, for the paranoid who are still reading and who don't want to add /Users, /Volumes, and /private to their shared Docker list, you can add:

/private/var/folders
/Users/<your username>/path/to/project_parent (e.g. /Users/jpangburn/Documents/vsc_workspaces/quantconnect_test where I have my QuantConnect projects under this dir)

Thanks for all your help Jasper, you led me right to this solution!  Really excited for local debugging :-) The online thing is good, but the debugging is tough on it when you're new and don't have a good feel for what objects have what properties.  The local debugger should make that a lot easier.

TLDR; for the easiest (but compromises on security) solution add the following to your Docker file sharing:

For the paranoid, add: