Share New Research 
As a Quantopian migrant, happy to finally have ported an algorithm to QC. It is a simple gap trade algorithm. Every day it narrows down the universe of US equites to :
price>10
previous day volume in top 15 percentile
It then takes 3 equities that gapped the most, checks their 5 day gain adds it, and selects top 2 to buy.
First impression of QC is that it is a much more robust platform. Obviously a ton of work went into creating it. Good job QC! On a negative side there are a few issues that might be a result of not knowing the platform well enough. Here there are:
a) It takes 16 seconds for backtest to execute on Quantopian, and 412 seconds on QuantConnect. And it is only a month long backtest.
b) Is there a simpler/more efficient way to achieve universe selection?
Every day I want to take US equities that have a price of more than $10, and are in top 15% by past 3 days trading volume. In Quantopian I would write something like:
pipe.set_screen(AverageDollarVolume(window_length=3, mask=(USEquityPricing.close.latest > 10)).percentile_between(85, 100))
What is the efficient way to do the same in QC?
c) How would one select by past X days of trading volume (the current algo does only previous day)?
d) Backtest console throws this error (but finished nevertheless)
Backtest Handled Error: We currently only support 500 symbols subscribed with minute resolution at a time due to physical memory limitations
What does this mean?
e) Are there plans for a debugger?
f) When I increased the test to 1 year I got this:
Runtime Error: System.Exception: Execution Security Error: Memory Usage Maxed Out - 8192MB max, with last sample of 9207MB.
Basically not being able to execute it.
h) Would I be able to run this algo live with 512MB limitation?
Currently my confidence of able to use QC is about 50%. I assume the algo is inefficient in some way and these concerns can be addressed by tweaking it.
Dan Whitnable
Vladimir, good to see you over here from Quantopian!
I have some of the same questions. One reason your algo might be taking so long is that it's running every minute (ie in the 'OnData' method. Maybe put that logic into a scheduled function to run at a specific time?
I'd also like to know if there are plans for a IDE debugger.
Good luck!
Petter Hansson
Didn't see this until now, maybe you found answers already. My 2 cents though:
a, b) Universe selection on QC is slow comparatively speaking. Depending on what you do you might be able to work around it, e.g. by doing different parts of the filtering in multiple passes.
c) You can probably calculate this by keeping around a dictionary for each symbol with RollingWindows as value where you add the daily volume. Then it's just a matter of summing the window contents. Yeah, ideally there should be some easier way for it.
d f) It puts an upper bound on your effective universe size, effectively. But as you noted, it's probably going to be too slow before 500 symbols.
h) Possibly. The live instances are fair less brittle because they don't get killed for spikes in memory consumption in the manner the backtests do. As long as your steady state memory use isn't too heavy, live will not be as problematic as backtests.
Jared Broad
Thank you for the comments Vladimir Prelovac - we're constantly working to make QC better, faster & stronger. We'll add these to our list and get to the ASAP. You might have noticed backtest load time dropped from 30s -> 9s on Thursday.
One key difference in our universe logic - we calculate everything "online" and its using identical code for backtesting and live trading. If we precache/synchronize data we can make it a few orders of magnitide faster (right now we're at the theoretical limit for CPU cycles). We've started the brain storming for pre-synchronizing all the data and will probably do it for minute data first(only).
Debugger's been mentioned a few times before in forum; high on our list but stability comes before features.
pipe.set_screen(AverageDollarVolume(window_length=3, mask=(USEquityPricing.close.latest > 10)).percentile_between(85, 100))
This would just be a linq statement on the CoarseFundamental object in the universe selection function.
We are going to dig into why python's consuming so much memory shortly after the bugs and IB extensions.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir Prelovac
Thanks for chiming in gentleman.
Is algorithm the problem or the limitations of the platform? Anything that can be done to speed up the algo as it is now? Would love to have a better understanding in 13 days or so left with Quantopian.
Thanks!
Jared Broad
Every minute you're doing a history request on an unspecified number of symbols. You should try to use rolling windows to build the history, limit the number of symbols through pre-filters, only do a history request each time you re-select the universe (vs every minute * 1500 symbols).
e.g. If you reselect the universe every week -- and use rolling windows for all the tradebars M-F; you'll eliminate 99% of the workload. Each history request is basically a whole other algorithm datafeed and you're running 1000+ of them each time you run a single loop.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vladimir Prelovac
Interesting, algorithm used to run, now the same algo throws this nasty error when cloned and backtested, what changed ?
9 | 16:38:52: Runtime Error: Python.Runtime.ConversionException: could not convert Python result to System.Object[] at Python.Runtime.Dispatcher.Dispatch (System.Collections.ArrayList args) [0x00018] in <69d4fd635645431aa1e955e701040fd1>:0 at __System_Func`2\[\[System_Collections_Generic_IEnumerable`1\[\[QuantConnect_Data_UniverseSelection_CoarseFundamental\, QuantConnect_Common\, Version=2_4_0_1\, Culture=neutral\, PublicKeyToken=null\]\]\, mscorlib\, Version=4_0_0_0\, Culture=neutral\, PublicKeyToken=b77a5c561934e089\]\,\[System_Object\[\]\, mscorlib\, Version=4_0_0_0\, Culture=neutral\, PublicKeyToken=b77a5c561934e089\]\]Dispatcher.Invoke (System.Collections.Generic.IEnumerable`1[T] ) [0x0000f] in <4f75b853bad54596a992f30aa4231859>:0 at QuantConnect.Algorithm.QCAlgorithm+<>c__DisplayClass243_0.<AddUniverse>b__0 (System.Collections.Generic.IEnumerable`1[T] c) [0x00000] in <922d0934d4f941f9aaa2a77de03e4c9d>:0 at QuantConnect.Data.UniverseSelection.CoarseFundamentalUniverse.SelectSymbols (System.DateTime utcTime, QuantConnect.Data.UniverseSelection.BaseDataCollection data) [0x00012] in <a8b01e3d761b4385be6efae9086f2317>:0 at QuantConnect.Data.UniverseSelection.Universe.PerformSelection (System.DateTime utcTime, QuantConnect.Data.UniverseSelection.BaseDataCollection data) [0x00001] in <a8b01e3d761b4385be6efae9086f2317>:0 at QuantConnect.Lean.Engine.DataFeeds.UniverseSelection.ApplyUniverseSelection (QuantConnect.Data.UniverseSelection.Universe universe, System.DateTime dateTimeUtc, QuantConnect.Data.UniverseSelection.BaseDataCollection universeData) [0x0024c] in <14785cb1084445cf8c93fdcb0adf454e>:0 at QuantConnect.Lean.Engine.DataFeeds.SubscriptionSynchronizer.Sync (System.DateTime frontier, System.Collections.Generic.IEnumerable`1[T] subscriptions, NodaTime.DateTimeZone sliceTimeZone, QuantConnect.Securities.CashBook cashBook, System.DateTime& nextFrontier) [0x00325] in <14785cb1084445cf8c93fdcb0adf454e>:0 at QuantConnect.Lean.Engine.DataFeeds.FileSystemDataFeed+<GetEnumerator>d__31.MoveNext () [0x000d6] in <14785cb1084445cf8c93fdcb0adf454e>:0 (Open Stacktrace)Dan Whitnable
Vladimir, the Universe selection method was changed in September 2017 (about the time of your original post). Take a look at the post below. Basically just need to return a Python list now and not a C# List. That may be the problem? Good luck.
Vladimir Prelovac
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!