Migrating from Quantopian to QuantConnect

The warmup feature is an improved historical data request. When we use it, most of the event handlers will be triggered with with historical data. If we just need warm up indicators, we can use History. However, SetWarmup is just one line of code. At the moment, Universe Selection doesn't work with warmup. We need to add symbols to the universe and make a History call in OnSecuritiesChanged method.

Lean/QuantConnect is data driven, so you need data before market open to trigger an event. By default, we subscribe to data that doesn't include extented market hours, but we can do it if we request it explicitly:

For the order of events please checkout the AlgorithmManager,Run method. You will see that OnData is the last method to run. Coarse/Fine Universe selection runs at midnight UTC.

"The warmup feature is an improved historical data request. When we use it, most of the event handlers will be triggered with with historical data. If we just need warm up indicators, we can use History. However, SetWarmup is just one line of code. At the moment, Universe Selection doesn't work with warmup. We need to add symbols to the universe and make a History call in OnSecuritiesChanged method."

Ok so this paragraph confused me. I did a little digging and found this example in the lean documentation (it's in C# but I'm assuming it will work similarly in python). I cloned it and I can see that it takes 100 days in a backtest of warming up before it places any trades.

My understanding:

I think i understand now that SetWarmup makes your algorithm "run" a specified number of days in the past, with most logic included. Which is good for things like a crossover that might happen once a month and you want to know if you already have a signal to buy without needing to wait for the next signal to occur. However, if you don't care about your algorithm's logic in the past but just need an indicator to tell you a value such as the 200 day SMA you can use history without SetWarmup. But none of that matters with Universe Selection, It can't look back into the past at all.

Are you saying that in live trading we would need to wait 100 days before any trades are placed if our universe selection uses a 100 day EMA? or is there a workaround?

Thanks for your patience,

Very good point LukeI - we're going to ponder on that one. You're 100% correct the existing setup won't handle it well. There are work arounds but they aren't clean. E.g. you can potentially request history in your selection function -- but you must be careful to request a small batch or you will timeout the request.  You should probably prefilter your universe by volume -- then apply the indicators (I'll try get you a tangible example shortly). 

We're going to give this some thought as it will probably need a light redesign and ultimately making Universe get triggered in warm up to avoid any manual history requests.

So while toying around I found something interesting. It seems that price for SPY may be incorrect? According to the logs:

2009-01-02 15:30:00 :75.702615744

2009-01-05 15:30:00 :78.043155393

2009-01-06 15:30:00 :77.883763804

2009-01-07 15:30:00 :78.420661788

2009-01-08 15:30:00 :76.021398922

2009-01-09 15:30:00 :76.348571131

2009-01-12 15:30:00 :74.670764931

2009-01-13 15:30:00 :72.942624545

When I look at historical charts SPY is trading in the low 90s high 80s in this timeframe. Any explination?

And to ask my original question why I did this. Is there any lookahead bias in the history function? Or does it get the current bar at that exact minute.

Thanks

Jonathan Gomez  - Data is split and dividend adjusted.

Thank you for the support guys! I have to imagine this shutdown of Quantopian has kept you busy. You've all been quick to answer quesions, and provides as much assistance in porting over algorithms. It is much appreciated! 

So, I finally completed the coversion of my algorithm! I just signed up for prime so I can backtest it first!

I apologize for this brute force questions as there has been a great discussion and multiple examples above on universe selection, but would it be of any use/value to users and the system to have a pre-defined high quality default universes? Before switching from quantopian, my systems had a heavy focus on starting with the Q500, Q1500, or Q3000(not documented yet) universes before filtering down to a much smaller set of assets.
 

From the Q1500 docs linked above:
default universe containing approximately 1500 US equities each day.

Constituents are chosen at the start of each month by selecting the top 1500 “tradeable” stocks by 200-day average dollar volume, capped at 30% of equities allocated to any single sector.

A stock is considered “tradeable” if it meets the following criteria:

1. The stock must be the primary share class for its company.
2. The company issuing the stock must have known market capitalization.
3. The stock must not be a depository receipt.
4. The stock must not be traded over the counter (OTC).
5. The stock must not be for a limited partnership.
6. The stock must have a known previous-day close price.
7. The stock must have had nonzero volume on the previous trading day.

While I see that some of the features of the Q1500 are easily repeatable here, I wonder about the more nuanced 'tradable' filters that make the assets appropriate for live trading with a higher degree of confidence. This makes using such universes nice as many bugs may be avoided due to the clean data. I would absolutely love to see a single line QC500, QC1500, QC3000 or other high quality universes cached in the system for quick backtesting/live trading.

Thank you AMDQuant !

Derek Tishler -- great idea and I like those filters. We already have a simple version of that in C# but hadn't brought it up yet as it has a bug in the cloud version of python. It looks like this:

This helper section could be expanded to include a QC500, 1500 with the extended filters. I've asked a team member to take a look into the python helper universe issue and will post back when its resolved.

That is another difference -- for QC history requests today's bar (or bars in general) won't be included in the history until the bar end time has passed. To have a partial bar for a current time step you would need to create a running tradebar for the day. This can be done with a Consolidator. 

@Jared

If today's last price isn't included in history, then how do you merge today's data with history to get an up-to-date moving average? If I was building a real time RSI-2 indicator, I would need to merge 1 day of history which would be yesterday's closing price, and the current price of the stock into a dataframe correct?

Which actually leads me to another question.

@Lukel 

 - You would need to merge the current day/minute/hour's bar to create the final one for your indicator if you want it dynamically updated intra-bar. It sounds like work.. the easiest way would probably be to create a rolling window and re-create the indicator each time.
 - Currect in QC anytime there's a bar count its refering to the market open bars so that code should be transportable.

In QC, is there an equivalent of Quantopian's data.can_trade before placing an order? Or is it not necessary in QC?

Hi guys, 

Thank you for helping us to switch from Qantopian to QC.

How can I follow the leverage ? On Q I was using:

If anyone is having trouble porting their code over from Quantopian, one thing that I discovered was causing a lot of errors was the whitespace. Quantopian didn't have a problem using 4 spaces as equivalent to tab, and mixing both methods but when you copy that code over QC has trouble reading it.

My solution was to go into a text editor like notepad++, enable "view tabs and whitespaces" and then change all space indents to tabs. That fixed a lot of problems that were hard to pin down. Also on QC the indents matter on blank lines, if you have a function with a blank line in it QC ends that function at the blank line. You need to indent your blank lines to the correct level so that QC will continue recognizing the rest of the function. These errors don't always get picked up in the text editor, you only see the real effect during the backtest with a mysterious error.

Thank you for mentioning it LukeI - if possible please send examples (support@quantconnect.com) as we're trying to create a code solution to the "white space issue" :) 

Chris D - We limit leverage for you like a real broker would do. To make it close to Quantopian's behavior you could set the leverage very high so you can trade without limits. Leverage for each security is set when you add the asset. 

Internally this is modelled as margin used. Each holding on leverage takes a cash position/margin to hold. At the time of purchase this is called "Initial Margin". To hold a position you need a "Maintenance Margin" fraction which is often lower than the initial margin. Margin is a good way of modelling multiple security types in a single portfolio as some assets have a fixed margin-requirement per contract (vs leverage which is variable on portfolio value).

Instead of following "total leverage" which only applies to a Equities-only portfolio of fixed leverage per asset; you should probably track TotalMarginUsed --> Portfolio.TotalMarginUsed. Perhaps one of the community or team will have a better idea here of what could be equivalent.

@Lukel, you could also use Ctrl+H and use the Replace All to replace all 4 consecutive spaces with a tab. This is what I do each time. Find/Replace is actually a very helpful functionality that I don't believe was available in the Quantopian IDE

Do you or any members here offer any algo coding service? Tutorials are a great effort but finding time to learn this all over again is problematic. I'd pay to have my algo ported.

ps. What if QC offered free algo rewrite for Quantopian members with a 1yr subscription to QC?

@Jared Broad - Thank you for your response. I will try to adapt this method.

Hi,

I'm also a migrant from Quantopian. I guess this forum is the one we migrants need. Please help us. I will learn from you guys and hopefully rebuild my algos here. :)

Thank you Vladimir its an interesting idea and appreciate the sentiment. We'll keep it in mind. Ideally we'd like to highlight the community members who can help with this.