Python Library To Run Quantopian Algorithm In Live

Quantopian — The Online Algo Trading Platform

Quantopian is one of the most popular online algo trading platforms and communities today. It provides the great backtesting environment where you can experiment with your idea, build algorithms and even participate in the contest, as well as share the idea and discuss it with smart people there.

Photo by  Rodion Kutsaev  on  Unsplash

One of the things many people have asked Alpaca during the beta program is how to run the algorithms that they built in Quantopian platform for their own purpose, not just for the contest. While Quantopian has built so much in the platform, they are so great to share the internal framework as open source zipline.

The Newest Open Source Libraries for Quantopian Users

Today, I wanted to share our newest open source libraries for Quantopian users; pylivetrader and pipeline-live.

alpacahq/pylivetrader
Python live trade execution library with zipline interface. - alpacahq/pylivetradergithub.com

alpacahq/pipeline-live
Pipeline Extension for Live Trading. Contribute to alpacahq/pipeline-live development by creating an account on GitHub.github.com

pylivetrader is a zipline API compatible trading framework in python which again focuses on live trading, with much less overhead and dependency problems. It is written from the ground up for live trading use cases, so it removes a lot of heavy lifting that zipline had to do such as price adjustment etc.

This means, you don’t need to build your data bundle to kick off your algorithm in live, but instead you can just start your live trading from the Quantopian algorithm source right away.

At the moment, the supported backend is only Alpaca, but we are happy to connect to IB etc. if someone contributes the code.

Pipeline API — the Core Piece of Quantopian Framework

Pipeline API is the core piece of Quantopian algorithm framework that allows easy stock selection based on the different metrics, much in a pythonic way, and this differentiates the platform from others. I found Pipeline is providing a tremendous value when it comes to trading wide range of universe. Unfortunately, it is not so easy for most people to use this great feature outside of the Quantopian platform.

pipeline-live is a python tool that allows you to do something similar anywhere so that you can do your research somewhere else as well as use it with existing python trading framework such as zipline-live or backtrader, including pylivetrader which I am introducing below. pipeline-live primarily uses IEX public API for pricing and basic fundamental information.

As you know, IEX provides market-wide volume data for daily OHLCV which makes it a perfect choice for pipeline usage. Since pipeline-live focuses on live trading use cases, it does not provide historical view unlike inside Quantopian, but the upside is it is fairly independent and easy to use. It is also very extensible so you can hook up with other paid data sources if you would find useful.

How to Convert Your Quantopian Algorithms to Run in Live Trading

We also put some practices together about how you could convert your Quantopian algorithms to run in live trading. You may want to take a look at these documents if you are interested in.

https://github.com/alpacahq/pipeline-live/blob/master/migration.md
https://github.com/alpacahq/pylivetrader/blob/master/migration.md

I also posted in Quantopian forum with the real example, and you may take a look at it, too.

Long-only non-day trading algorithm for live
This is a modified version of the algorithm presented in…www.quantopian.com

Feel free to give me any feedback/questions/criticism. Happy to help you get started with live trading with these tools too.

And here is the example code migrated from the post above.


/

I'm a College Student, and I'm Still Building my Robo-Advisor

All code and outside instructions for setup, etc. can be found in the GitHub repository here. This is a post is a sequel to building a robo advisor in Quantopian. You can find the first part on Hackernoon here, or on Alpaca’s blog here.

I’m…still…working…

I’m…still…working…

Hey all, I’m Rao, an intern at Alpaca. About two weeks ago, I published an article on HackerNoon about the Robo-Advisor I’ve been working on. In the last two weeks, I’ve moved my work offline, pushing development with Zipline. In that time, I’ve picked up a few tips and tricks, and I wanted to share that with you.

Zipline — What, When, Why, Where, How?

We’ve now created a functional robo advisor on Quantopian. But, the IDE puts limits on the scale and sophistication of our work. To extend this robo advisor further, it’s best to do our future work locally with Zipline.

This shift offers advantages. We can use multiple files, configuration files, and generally develop out robo-advisor like any other python app. Being free from a single file format means that our work can be more organized, and individual functions less bloated (more on that later).

But, it’s not all roses. Developing in Zipline comes with its own challenges. But, if I can do it, you can too!

Installing zipline on your local machine isn’t as simple as `pip install zipline`. It has several dependencies, which vary slightly for each OS. You can find all necessary installation instructions here.

I made several mistakes during installation, so I recommend you run zipline in either a virtual environment, or my personal preference, in a Docker container. I’ve attached both the finished container to play with, as well as the corresponding Dockerfile. Take some time, figure out your development environment. Ready? Let’s get started.

Zipline Commands — Bundles and Execution

Open a new python file locally. Copy and paste your code from the Quantopian IDE. Now, we’re ready to go- just yanking you around. We’re not close to ready. Quantopian’s IDE is a great development environment. It takes care of background processes, allowing you to focus solely on your algorithm.

First, we need historical data to run our algorithm on. Zipline reads through data using something called bundles. To use the bundles, they need to be ingested first, which is done using the command:

$zipline ingest -b <bundle>

Zipline provides two bundles, quandl and quantopian-quandl. To ingest quandl, you’ll need to make an account, and then obtain an API key. You won’t need to for quantopian-quandl. Zipline also provides the option to create custom bundles, by writing a custom ingest function. That will be reserved for a later post.

Both Quandl and quantopian-quandl don’t provide ETF data, so I’ve provided a custom alpaca bundle. The instructions for installing this bundle locally can be found in the GitHub repository readme.

You run zipline code using the following format:

$zipline run -f <filename> -b <bundle> --start <date> --end <date>

You can see a lot of the GUI input from Quantopian is configured from the command line with Zipline.

Docker Containers:

I’ve created a docker container running python 3.6 with zipline and necessary dependencies installed. If you’re hesitant about installing zipline locally, you can pull the container from the docker hub, and experiment with the environment.

Run the image with the command:

$docker run -it alpaca/roboadvisor /bin/bash

A First Zipline Example — Buy and Hold

Like my initial start with Quantopian, I started with Zipline using a simple buy and hold strategy.

Here are a few differences to keep in mind. First, all the custom functions like order, symbols, etc. are no longer automatically included, and have to be manually imported (from zipline.api import *). Next, this is rather silly, but now that we’re not in Quantopian’s IDE, we’re no longer going to use log.info to track transactions, but print statements.

Let’s run the code with the format from earlier, and run the tests from January to June:

$zipline run -f buy-and-hold.py -b alpaca --start 2018-01-01 --end 2018-06-01

If there’re no syntax errors, Zipline will spit out a whole bunch of data. How do we know if we’re right? Scroll until you find the following table from STDOUT:

Look at the cumulative alpha value from the last simulated trading day (2018–06–01)

Look at the cumulative alpha value from the last simulated trading day (2018–06–01)

Find the cumulative alpha value (performance against benchmark), and compare it to the cumulative alpha value when you run the same algorithm on Quantopian. They should be very similar values.

We’ve got a good idea of how Zipline works, so we can go ahead and implement the rest of our single-universe algorithm (distance computation and rebalancing). I spent an entire post talking about this implementation, so I don’t think there’s a need to re-hash it. If you haven’t read that, I recommend you do — it’s a great read! (a totally unbiased opinion)

Multiple Universes

In the previous post, I showed how to expand the algorithm to cover all possible Vanguard universes by adding multiple dictionaries in initialize function. But, if we want to implement all six vanguard universes, we would finish with a bloated ingest function, which doesn’t look that pretty.

But now we’re away from Quantopian, and here with Zipline! We can spread our code across multiple files, so let’s go ahead and do that.

To be used by the algorithm, a universe needs two sets of information. The first is the set of symbols, and the second is the weight distribution based on the risk. From the way that the symbols function works, each universe’s list of symbols needs to set in the initialization function.

But, the weights are dictionaries that aren’t bound to any zipline.api functions, so we can actually configure those in a separate file. The ConfigParser package can read in data stored in an external .ini file. More importantly, it stores the data in a similar structure to a dictionary (key/value), and is easily called and organized.

First, we need to install the package:

$pip3 install ConfigParser

With the package installed, it’s time to create the INI file. The INI file contains the information for allocation based on risk level.

INI files are separated into sections, with a new section delineated by a section title in brackets. The section titles act much like a key in dictionaries, and is a useful way to organize inputs. This file organized each individual Vanguard universe as its own section. If you’d like to add your own universe, fork the gist, create a new section with the universe name in brackets, and list the information below it.

The robo-advisor’s algorithm parses input as a dictionary. While the INI file works much like a dictionary, we’ll still need to write a function to actually translate it into a dictionary.

def section_to_dict(section):
    config = ConfigParser()
    config.read('universe-config.ini')
    out_dict = {}
    for key in config[section]:
        out_dict[int(key)] = ast.literal_eval(config[section][key])
    return(out_dict)

The configparser is initialized, and then reads the given INI file. The section that’s read is given as user input. If we refer back to the INI file, we’ll see that the values take the form as key = value. So like a dictionary, we can iterate through the deys of an INI section. In each case, each key/value pair is added to the dictionary. (Note: If you receive a key error, it’s because the path to your config.ini file is wrong

Two more things to be aware of. In the INI file, everything is a string, but our algorithm expects an integer/tuple key-value pairs. For the key, just casting them as integers is enough, but for the tuple, we’ll have to unstring the value. For that, I found ast.literal_evalto be the function that worked best.

Now, let’s integrate this new method of retrieving weight-based allocation with our robo-advisor algorithm:

I’ve listed all the symbols for every universe in the initialization. The weights for that universe are determined by calling section_to_dict on the appropriate section of the INI file.

Now, feel free to add as many universes as you like!

What’s Next?

Going from here, I’m interested in visualizing all the output data from the Zipline backtest. With my next post, I’m hoping to explore the different visualizations available from that raw data.

by Rao Vinnakota

/

I Built a Go Plugin for Alpaca’s MarketStore as a College Intern

Hey all! I’m Ethan and recently started working for Alpaca as a Software Engineering Intern! For my first task, I created a Go plugin for Alpaca’s open source MarketStore server that fetches and writes Binance minute-level.

Image from iOS (2).jpg

You might be wondering — What is MarketStore? MarketStore is a database server written in Go that helps users handle large amounts of financial data. Inside of MarketStore, there are Go plugins that allow users to gather important financial and crypto data from third party sources.

For this blog post, I’ll be going over how I created the plugin from start to finish in three sections: Installing MarketStore, understanding MarketStore’s plugin structure, creating the Go plugin., and installing the Go plugin.

Experience Installing and Running MarketStore Locally

First, I set up MarketStore locally. I installed the latest version of Go and started going through the installation process outlined in MarketStore’s README. All the installation commands worked swimmingly, but when I tried to run marketstore using

ethanc@ethanc-Inspiron-5559:~/go/bin/src/github.com/alpacahq/marketstore$ marketstore -config mkts.yml

I got this weird error:

/usr/local/go/src/fmt/print.go:597:CreateFile/go/src/github.com/alpacahq/marketstore/executor/wal.go:87open /project/data/mktsdb/WALFile.1529203211246361858.walfile: no such file or directory: Error Creating WAL File

I was super confused and couldn’t find any other examples of this error online. After checking and changing permissions in the directory, I realized my mkts.yml file configuration root_directory was incorrect. To resolve this, I changed mkts.yml from

root_directory: /project/data/mktsdb

To

root_directory: /home/ethanc/go/bin/src/github.com/alpacahq/marketstore/project/data/mktsdb

and reran

ethanc@ethanc-Inspiron-5559:~/go/bin/src/github.com/alpacahq/marketstore$ marketstore -config mkts.yml

This time, everything worked fine and I got this output:

ethanc@ethanc-Inspiron-5559:~/go/bin/src/github.com/alpacahq/marketstore$ marketstore -config mkts.yml
…
I0621 11:37:52.067803 27660 log.go:14] Launching heartbeat service…
I0621 11:37:52.067856 27660 log.go:14] Enabling Query Access…
I0621 11:37:52.067936 27660 log.go:14] Launching tcp listener for all services
…

To enable the gdaxfeeder plugin which grabs data from a specified cryptocurrency, I uncommented these lines in the mkts.yml file:

and reran

ethanc@ethanc-Inspiron-5559:~$ marketstore -config mkts.yml

which yielded:

…
I0621 11:44:27.248433 28089 log.go:14] Enabling Query Access…
I0621 11:44:27.248448 28089 log.go:14] Launching tcp listener for all services…
I0621 11:44:27.254118 28089 gdaxfeeder.go:123] lastTimestamp for BTC = 2017–09–01 04:59:00 +0000 UTC
I0621 11:44:27.254189 28089 gdaxfeeder.go:123] lastTimestamp for ETH = 0001–01–01 00:00:00 +0000 UTC
I0621 11:44:27.254242 28089 gdaxfeeder.go:123] lastTimestamp for LTC = 0001–01–01 00:00:00 +0000 UTC
I0621 11:44:27.254266 28089 gdaxfeeder.go:123] lastTimestamp for BCH = 0001–01–01 00:00:00 +0000 UTC
I0621 11:44:27.254283 28089 gdaxfeeder.go:144] Requesting BTC 2017–09–01 04:59:00 +0000 UTC — 2017–09–01 09:59:00 +0000 UTC
…

Now that I got MarketStore running, I used Jupyter notebooks and tested out the commands listed in this Alpaca tutorial and got the same results. You can read more about how to run MarketStore in MarketStore’s README, Alpaca’s tutorial, and this thread.

Understanding how MarketStore Plugins work

After installing, I wanted to understand how their MarketStore repository works and how their current Go plugins work. Before working in Alpaca, I didn’t have any experience with the Go programming language. So, I completed the Go’s “A Tour of Go” tutorial to get a general feel of the language. Having some experience with C++ and Python, I saw a lot of similarities and found that it wasn’t as difficult as I thought it would be.

Creating a MarketStore Plugin

To get started, I read the MarketStore Plugin README. To summarize at a very high level, there are two critical Go features which power plugins: Triggers and BgWorkers. You use triggers when you want your plugin to respond when certain types data are written to your MarketStore’s database. You would use BgWorkers if you want your plugin to run in the background.

I only needed to use the BgWorker feature because my plugin’s goal is to collect data outlined by the user in the mkts.yml configuration file.

To get started, I read the code from the gdaxfeeder plugin which is quite similar to what I wanted to do except that I’m trying to get and write data from the Binance exchange instead of the GDAX exchange.

I noticed that the gdaxfeeder used a GDAX Go Wrapper, which got its historical price data public endpoint. Luckily, I found a Go Wrapper for Binance created by adshao that has the endpoints which retrieves the current supported symbols as well as retrieves Open, High, Low, Close, Volume data for any timespan, duration, or symbol(s) set as the parameters.

To get started, I first created a folder called binancefeeder then created a file called binancefeeder.go inside of that. I then first tested the Go Wrapper for Binanceto see how to create a client and talk to the Binance API’s Kline endpoint to get data:

I then ran this command in my root directory:

ethanc@ethanc-Inspiron-5559:~/go/bin/src/github.com/alpacahq/marketstore$ go run binancefeeder.go

and received the following response with Binance data:

&{1529553060000 6769.28000000 6773.91000000 6769.17000000 6771.34000000 32.95342700 1529553119999 223100.99470354 68 20.58056800 139345.00899491}
&{1529553120000 6771.33000000 6774.00000000 6769.66000000 6774.00000000 36.43794400 1529553179999 246732.39415947 93 20.42194600 138288.41850603}
…

So, it turns out that the Go Wrapper worked!

Next, I started brainstorming how I wanted to configure the Binance Go plugin. I ultimately chose symbols, queryStart, queryEnd, and baseTimeframe as my parameters since I wanted the user to query any specific symbol(s), start time, end time, and timespan (ex: 1min). Then, right after my imports, I started creating the necessary configurations and structure for BinanceFetcher for a MarketStore plugin:

The FetcherConfig’s members are what types of settings the user can configure in their configuration file (ex: mkts.yml) to start the plugin. The BinanceFetcher’’s members are similar to the FetcherConfig with the addition of the config member. This will be used in the Run function later.

After creating those structures, I started to write the background worker function. To set it up, I created the necessary variables inside the backgroundworker function and copied the recast function from the gdaxfeeder. The recast function uses Go’s Marshal function to encode the config JSON data received, then sets a variable ret to an empty interface called FetcherConfig. Then it stores the parsed JSON config data in the ret variable and returns it:

Then inside the NewBgWorker function, I started to create a function to determine and return the correct time format as well as set up the symbols, end time, start time, and time duration. If there are no symbols set, by default, the background worker retrieves all the valid cryptocurrencies and sets the symbol member to all those currencies. It also checks the given times and duration and sets them to defaults if empty. At the end, it returns the pointer to BinanceFetcher as the bgworker.BgWorker:

Then, I started creating the Run function which is implemented by BgWorker (see bgworker.go for more details). To get a better sense of how to handle errors and write modular code in Go, I read the code for plugins gdaxfeeder and polygon plugins. The Run function receives the BinanceFetcher (which is dereferenced since bgworker.BgWorker was the pointer to BinanceFetcher). Our goal for the Run function is to call the Binance API’s endpoint with the given parameters for OHLCV and retrieve the data and writes it to your MarketStore’s database.

I first created a new Binance client with no API key or secret since I’m using their API’s public endpoints.

Then, to make sure that the BinanceFetcher doesn’t make any incorrectly formatted API calls, I created a function to check the timestamp format using regex and change it to the correct one. I had to convert the user’s given timestamp to maintain consistency in the Alpaca’s utils.Timeframe which has a lot of helpful functions but has different structure members than the one’s Binance uses (ex: “1min” vs. “1m”). If the user uses an unrecognizable timestamp format, it sets the baseTimeframe value to 1 minute:

The start and end time objects are already checked in the NewBgWorker function and returns a null time.Time object if invalid. So, I only have to check if the start time is empty and set it to the default string of the current Time. The end time isn’t checked since it will be ignored if incorrect which will be explained in the later section:

Now that the BinanceFetcher checks for the validity of its parameters and sets it to defaults if its not valid, I moved onto programming a way to call the Binance API. 

To make sure we don’t overcall the Binance API and get IP banned, I used a for loop to get the data in intervals. I created a timeStart variable which is first set to the given time start and then created a timeEnd variable which is 300 times the duration plus the timeStart's time. At the beginning of each loop after the first one, the timeStart variable is set to timeEnd and the timeEnd variable is set to 300 times the duration plus the timeStart’s time:

When it reaches the end time given by the user, it simply alerts the user through glog and continues onward. Since this is a background worker, it needs to continue to work in the background. Then it writes the data retrieved to the MarketStore database. If invalid, the plugin will stop because I don’t want to write garbage values to the database:

Installing Go Plugin

To install, I simply changed back to the root directory and ran:

ethanc@ethanc-Inspiron-5559:~/go/bin/src/github.com/alpacahq/marketstore$ make plugins

Then, to configure MarketStore to use my file, I changed my config file, mkts.yml, to the following:

Then, I ran MarketStore:

ethanc@ethanc-Inspiron-5559:~/go/bin/src/github.com/alpacahq/marketstore$ marketstore -config mkts.yml

And got the following:

…
I0621 14:48:46.944709 6391 plugins.go:42] InitializeBgWorkers
I0621 14:48:46.944801 6391 plugins.go:45] bgWorkerSetting = &{binancefeeder.so BinanceFetcher map[base_timeframe:1Min query_start:2018–01–01 00:00 query_end:2018–01–02 00:00 symbols:[ETH]]}
I0621 14:48:46.952424 6391 log.go:14] Trying to load module from path: /home/ethanc/go/bin/bin/binancefeeder.so…
I0621 14:48:47.650619 6391 log.go:14] Success loading module /home/ethanc/go/bin/bin/binancefeeder.so.
I0621 14:48:47.651571 6391 plugins.go:51] Start running BgWorker BinanceFetcher…
I0621 14:48:47.651633 6391 log.go:14] Launching heartbeat service…
I0621 14:48:47.651679 6391 log.go:14] Enabling Query Access…
I0621 14:48:47.651749 6391 log.go:14] Launching tcp listener for all services…
I0621 14:48:47.654961 6391 binancefeeder.go:198] Requesting ETH 2018–01–01 00:00:00 +0000 UTC — 2018–01–01 05:00:00 +0000 UTC
…

Testing:

When I was editing my plugin and debugging, I often ran the binancefeeder.go file:

ethanc@ethanc-Inspiron-5559:~/go/bin/src/github.com/alpacahq/marketstore$ go run binancefeeder.go

If I ran into an issue I couldn’t resolve, I used the equivalent print function for Go (fmt). If there is an issue while running the plugin as part of MarketStore via the marketstore -config mkts.yml command, I used the glog.Infof() or glog.Errorf() function to output the corresponding error or incorrect data value.

Lastly, I copied the gdaxfeeder test go program and simply modified it for my binancefeeder test go program.

You’ve made it to the end of the blog post! Here’s the link to the Binance plugin if you want to see the complete code. If you want to see all of MarketStore’s plugins, check out this folder.

To summarize, if you want to create a Go extension for any open source repository, I would first read the existing documentation whether it is a README.md or a dedicated documentation website. Then, I would experiment around the repositories code by changing certain parts of the code and see which functions correspond with what action. Lastly, I would look over previous extensions and refactor an existing one that seems close to your plugin idea.

Thanks for reading! I hope you take a look at the MarketStore repository and test it out. If you have any questions, few free to comment below and I’ll try to answer!

Special thanks to Hitoshi, Sho, Chris, and the rest of the Alpaca’s Engineering team for their code reviews and help as well as Yoshi and Rao for providing feedback for this post.

By: Ethan Chiu

/

I am a College Student and I Built My Own Robo Advisor

I’m Rao, and I’m an intern at Alpaca working on building an open-source robo advisor. I don’t have much experience in the space, and I had to find a lot of answers. While there’s a wealth of material available on the web, very little is organized. This post is the combination of various threads and forums that I read through looking for answers.

“Yes, you an do it too!”

“Yes, you an do it too!”

What is a Robo Advisor anyway?!

Robo advisors are automated advising services that require little to no user interaction. They specialize in maintaining portfolios based on the investors chosen risk level. Btw, they were launched at the start of the financial crisis in 2008.

The actual logic for a robo advisor is straightforward.

  • Allocation” — Given a risk level, portions of capital are allocated to different positions.
  • Distance” — Over a regular time interval, the adviser scans to see if there’s a significant change in the portfolio balance, and
  • Rebalancing” — If necessary, rebalances.

Allocation:

The process of allocation is where different weights are assigned to asset classes based on the developer’s algorithm. These asset classes cover the various aspects of the market, and keep a diversified portfolio. Wealthfront sees these asset classes as:

  • US Stocks
  • Foreign Stocks
  • US Bonds
  • Foreign Bonds
  • Inflation Protection

Each of these assets is represented with ETFs, who capture a broad look at the asset class.

Distance:

Rebalancing a portfolio comes with a certain amount of overhead and cost. To avoid this, the robo advisor computes a “distance” vector. To trigger a rebalance, this distance will have to be greater than a given threshold. Most robo advisors usually have this threshold set to 5 percent.

The distance in this case, is the individual change in weight of each position. This is done by first calculating the current allocation. This is the value of a position against the total value of the portfolio. If any one position has a weight that’s 5 percent greater (current_weight/target_weight > 1.05), then the robo advisor will trigger a rebalance

Rebalancing:

Rebalancing is the act of buying and selling to return the portfolio to its target state. It’s important that all selling is done first. Once all cash available is gathered, necessary purchases are made.

To calculate how much to sell, use the target allocation of cash (weight * portfolio value), and see the number of shares that result from that cash. The difference between the number of shares currently held and the target level of shares is the amount to buy/sell.

Implementing a Robo Advisor with the Vanguard Core Series:

Now that we’ve established the various parts of what makes a robo advisor, we’ll start executing the steps to build one. We’ll build our robo advisor using Quantopian’s IDE.

When implementing our advisor, I find it easiest to do in steps. Quantopian’s IDE can only run backtests on a full algorithm. What that means, is there’s no “halfway” development. So, each step is a self-contained algorithm by itself.

Note: The implementation of Vanguard’s Core Series is using the information found here. The portfolio includes a 2% investment in a money market fund, but that isn’t available on Quantopian, so for the purposes of this tutorial, we’ll ignore it.

Risk-Based Allocation:

The first step, is to determine our universe, and assign weights using risk level. Our universe will be the Vanguard Core series:

  • VTI — Domestic Equity
  • VXUS — International Equity
  • BND — Domestic Fixed Income (Bonds)
  • BNDX — International Fixed Income (Bonds)

The risk level is actually the ratio of fixed income to equity, with equity being more volatile. So, a risk level 0 would have all capital allocated to fixed income (BND, BNDX), while a risk level of 5 would be 50/50. All 11 possible risk levels are catalogued in a dictionary, with the keys being the risk level, and values being tuples containing weight allocation:

risk_level = 5
risk_based_allocation = {0: (0,0,0.686,0.294),
                         1: (0.059,0.039,0.617,0.265),
                         2: (0.118,0.078,0.549,0.235),
                         3: (0.176,0.118,0.480,0.206),
                         4: (0.235,0.157,0.412,0.176),
                         5: (0.294,0.196,0.343,0.147), 
                         6: (0.353,0.235,0.274,0.118),
                         7: (0.412,0.274,0.206,0.088),
                         8: (0.470,0.314,0.137,0.059),
                         9: (0.529,0.353,0.069,0.029),
                         10: (0.588,0.392,0,0)}

The next step will be to implement our allocation in Quantopian. For this step, the only methods we’ll need are initialize and handle_data. The rest are superfluous.

The initialize method is the main. When a backtest is started, it automatically calls initialize. So, initialize will be the function that contains the dictionary for risk levels, as well as choosing the risk levels. The handle_data method is called automatically after initialize. It’s here that we’ll actually purchase the positions outlined in initialize.

Both initialize and handle_data have the variables context and data. Context allows the user to store global variables that get passed from method to method. Data helps the algorithm fetch different sorts of data. (Note: This is a change from Quantopian 1 where data was the object where global variables were stored).

Let’s get started. In initialize, copy and paste the following code:

context.stocks = symbols(‘VTI’, ‘VXUS’, ‘BND’, ‘BNDX’)
context.bought = False

risk_level = 5
risk_based_allocation = {0: (0,0,0.686,0.294),
                         1: (0.059,0.039,0.617,0.265),
                         2: (0.118,0.078,0.549,0.235),
                         3: (0.176,0.118,0.480,0.206),
                         4: (0.235,0.157,0.412,0.176),
                         5: (0.294,0.196,0.343,0.147),
                         6: (0.353,0.235,0.274,0.118),
                         7: (0.412,0.274,0.206,0.088),
                         8: (0.470,0.314,0.137,0.059),
                         9: (0.529,0.353,0.069,0.029),
                         10: (0.588,0.392,0,0)}
    #Saves the weights to easily access during rebalance
context.target_allocation = dict(zip(context.stocks,
                            risk_based_allocation[risk_level]))

    #To make initial purchase
context.bought = False

The variable context.stocks is a list of stock objects. The symbols function turns the strings into objects. The objects have attributes like current price, closing price, etc. Using that list, as well as the dictionary for allocation, we’ll create a second dictionary context.target_allocation. The keys are each ticker (VTI, VXUS, etc.) and the values are the ticker’s weights. This dictionary will be useful for both allocation and rebalancing.

Copy and paste the following code to create handle_data:

if not context.bought:
        for stock in context.stocks:
            amount = (context.target_allocation[stock] *
                      context.portfolio.cash) / data.current(stock,’price’)
            if (amount != 0):
                order(stock, int(amount))
                #log purchase
            log.info(“buying “ + str(int(amount)) + “ shares of “ +
                     str(stock))

         #now won’t purchase again and again
         context.bought = True

The variable context.bought refers to the value originally set to False in initialize. Since handle_data is called every time a market event occurs. So, using context.bought ensures the shares are only bought once.

To buy shares for each stock, the list of stock objects is iterated through. For each stock, the total number of shares are calculated by allocating the proper amount of capital (weight * capital), and then dividing that capital by the current price. Since we can only buy whole shares, leftover capital is added back to the cash on hand.

Lastly, for a smoother backtesting experience, all transactions are logged. The log is in the lower right hand corner. Build the current algorithm. Set the start date to June 2013 or later as the ETF BNDX was first tradeable on June 7, 2013.

Calculating Distance:

Before rebalancing a portfolio, it’s best to first calculate if the current portfolio is worth rebalancing. In more sophisticated robo advisors, this step would have a target portfolio calculated with an algorithm. For this example, our target portfolio is going to be the initial allocation defined by the risk level in initialize.

As the value of each position fluctuates, their portfolio weight may change too. The distance vector will be the change in weight of each individual position. Calculating the distance takes place in three steps:

  1. Calculating the current weights of each position
  2. Calculating the target weights (already done)
  3. Comparing the two weights

We want to check the balance daily, so we’ll go ahead and add the following function , before_trading_starts, to our code:

def before_trading_starts(context, data):
    #total value
    value = context.portfolio.portfolio_value +
                                         context.portfolio.cash
    #calculating current weights for each position
    for stock in context.stocks:
        if (context.target_allocation[stock] == 0):
            continue
        current_holdings = data.current(stock,’close’) *
                      context.portfolio.positions[stock].amount
        weight = current_holdings/value
        growth = float(weight) /
                        float(context.target_allocation[stock])
        if (growth >= 1.025 or growth <= 0.975):
            log.info(“Need to rebalance portfolio”)
            break

We first calculate total value. Since that value doesn’t change, it can be down outside the loop. Then, each position’s individual weight is calculated, and compared to the target weight (we’re using division since we’re interested in the relative growth of the position, not the absolute). If that growth exceeds the threshold (currently 2.5%) or is under the threshold (de-valued), then a rebalance is triggered. We don’t have an actual rebalance function written, so for now, we’ll simply log that a rebalance is necessary. It’s important to note the break added. Once the algorithm realizes that it’s time to rebalance, there’s no need to continue checking the rest of the stock. It creates a better best-case scenario.

However, we’re not actually done. We have to call before_trading_start, but calling it in a loop is inefficient. That’s why we’ll use the schedule_function command in initialize. Add this line of code to the end of the function block:

    schedule_function(
    func=before_trading_starts,
    date_rule=date_rules.every_day(),
    time_rule=time_rules.market_open(hours=1))

This schedules a distance calculation as the market opens every day. By computing the distance in the morning, we have the time and space necessary to execute rebalancing actions.

Rebalancing

The act of rebalancing a portfolio occurs in two steps. First, all assets that need to be sold are sold, and then all assets that need to bought are bought. Selling first makes sure that we don’t run out of cash.

The following is the code for a rebalance function:

def rebalance(context, data):
    for stock in context.stocks:
        current_weight = (data.current(stock, ‘close’) *
                          context.portfolio.positions[stock].amount) /
                          context.portfolio.portfolio_value
        target_weight = context.target_allocation[stock]
        distance = current_weight — target_weight
        if (distance > 0):
            amount = -1 * (distance * context.portfolio.portfolio_value) /
                     data.current(stock,’close’)
            if (int(amount) == 0):
                continue
            log.info(“Selling “ + str(int(amount)) + “ shares of “ +
                      str(stock))
            order(stock, int(amount))
    for stock in context.stocks:
        current_weight = (data.current(stock, ‘close’) *
                          context.portfolio.positions[stock].amount) /
                          context.portfolio.portfolio_value
        target_weight = context.target_allocation[stock]
        distance = current_weight — target_weight
        if (distance < 0):
            amount = -1 * (distance * context.portfolio.portfolio_value) /
                     data.current(stock,’close’)
            if (int(amount) == 0):
                continue
            log.info(“Buying “ + str(int(amount)) + “ shares of “ +
                      str(stock))
            order(stock, int(amount))

The selling and buying mechanics are the same. The only difference is when selling stock, you use a negative value when calling the order function. In both cases, we take the absolute difference in weight (target — current), and use that value to find the number of shares to buy or sell.

Multiple Universes

Vanguard provides several more universes beyond the core-series. Currently, we’re able to manipulate the risk level, and observe the outcome. Let’s add the functionality of also being able to select a universe.

The first and straightforward method, is to implement multiple dictionaries. Here’s how something like that would look. The CRSP series was added to our algorithm. The user now chooses both the universe and the risk level.

    core_series = symbols(‘VTI’, ‘VXUS’, ‘BND’, ‘BNDX’)
    crsp_series = symbols(‘VUG’, ‘VTV’, ‘VB’, ‘VEA’, ‘VWO’, ‘BSV’, ‘BIV’, ‘BLV’, ‘VMBS’, ‘BNDX’)
    #universe risk based allocation
    core_series_weights = {0: (0,0,0.686,0.294),
                           1: (0.059,0.039,0.617,0.265),
                           2: (0.118,0.078,0.549,0.235),
                           3: (0.176,0.118,0.480,0.206),
                           4: (0.235,0.157,0.412,0.176),
                           5: (0.294,0.196,0.343,0.147),
                           6: (0.353,0.235,0.274,0.118),
                           7: (0.412,0.274,0.206,0.088),
                           8: (0.470,0.314,0.137,0.059),
                           9: (0.529,0.353,0.069,0.029),
                           10: (0.588,0.392,0,0)}
    crsp_series_weights = {0: (0,0,0,0,0,0.273,0.14,0.123,0.15,0.294),
1: (0.024,0.027,0.008,0.03,0.009,0.245,0.126,0.111,0.135,0.265),
2: (0.048,0.054,0.016,0.061,0.017,0.218,0.112,0.099,0.12,0.235),
3: (0.072,0.082,0.022,0.091,0.027,0.191,0.098,0.086,0.105,0.206),
4: (0.096,0.109,0.03,0.122,0.035,0.164,0.084,0.074,0.09,0.176),
5: (0.120,0.136,0.038,0.152,0.044,0.126,0.07,0.062,0.075,0.147),
6: (0.143,0.163,0.047,0.182,0.053,0.109,0.056,0.049,0.06,0.118),
7: (0.167,0.190,0.055,0.213,0.061,0.082,0.042,0.037,0.045,0.088),
8: (0.191,0.217,0.062,0.243,0.071,0.055,0.028,0.024,0.030,0.059),
9: (0.215,0.245,0.069,0.274,0.079,0.027,0.014,0.013,0.015,0.029),
10: (0.239,0.272,0.077,0.304,0.088,0,0,0,0,0)}

    #set universe and risk level
    context.stocks = crsp_series
    risk_based_allocation = crsp_series_weights
    risk_level = 1

The user can use the three variables, context.stocks, risk_based_allocation, and risk_level, to set both the universe and the risk level.

What’s Next

Developing in Quantopian is a great experience — they provide lots of useful tools. But, it’s also limiting, being forced to only work in one file, etc. To build further on this robo advisor, I plan to migrate my work so far to the backend, and use Quantopian’s open source python library Zipline to run backtests. The next installment will cover all this and more!

You can find the algorithm here. Try it yourself!

by Rao Vinnakota

/

How to Setup Bitcoin Historical Price Data for Algo Trading in Five Minutes

Alpaca platform is now accepting signups for our algo trading waitlist and we do believe trading automation is the future. 

Photo by  Chris Liverani  on  Unsplash

One of the great benefits of algorithmic trading is that you can test your trading strategy against historical data.

Especially for new strategies you developed on your own, you don’t really know how it will perform without testing it against reliable data. Algo trading is not that different from software development. Today, most good software code is continuously tested. As the code evolves, developers continually test it against real use cases to make sure that alterations won’t result in future failures. You want to test your algo when you drop the very first version. But you also want to test the algo when you make changes or adjustments while the algo is running.

Here is one problem you face; data. 

While it is always good to test as many angles as possible, running a number of backtests is going to be a very data-intensive workload that requires access to enough data to have visibility into a very long history. This is why you cannot run this iteration using GDAX API directly. You need to store the data somewhere for your own purposes.

The Tutorial

So the first question to come to mind is always how to get the data and prepare it for successful backtesting. Well, today I am going to tell you how to use MarketStore to acquire a long history of Bitcoin price data for this purpose of running the most accurate backtest possible. And this setup tutorial is going to be quick. Don’t waste your time setting up this and that. All you need is:

  • docker (either on Windows, Mac or Linux),

  • a console terminal,

  • and a big cup of strong coffee! ☕

Got’em? Alright, let’s get started.

 

Architecture

Medium - bitcoin data1.png

Here is the high level picture of today’s system. We will start a MarketStore instance using docker container, and run a background worker that calls GDAX price API so that we can pull the bitcoin historical price from their endpoint quickly and make it available for backtest clients to query over HTTP.

We will start another container for the client using python anaconda with python3 image. We use the official client package named pymarkestore. You will get a DataFrame from MarketStore.

Setup MarkeStore Server

There is the official build of MarketStore docker image today publicly available in DockerHub, but first, let’s write a config file for the server.

In the github repository you can find an example config file in YAML format: https://github.com/alpacahq/marketstore/blob/master/mkts.yml but I’m putting our example here.

root_directory: /project/data/mktsdb   listen_port: 5993   log_level: info   queryable: true   stop_grace_period: 0   wal_rotate_interval: 5   enable_add: true   enable_remove: false   enable_last_known: false   bgworkers:     - module: gdaxfeeder.so       name: GdaxFetcher       config:         symbols:          - BTC          base_timeframe: "1D"          query_start: "2018-01-01 00:00"

This configures the server so that it fetches the GDAX historical price API for 1-day bars since 2018–01–01. Save this config as $PWD/mkts.yml file. The server listens on the port 5993 as default. Now let’s bring up the server.

$ docker run -v $PWD/mktsdb:/project/data/mktsdb -v $PWD/mkts.yml:/tmp/mkts.yml --net host alpacamarkets/marketstore:v2.1.1 marketstore -config /tmp/mkts.yml

The server should automatically download the docker images from DockerHub if you haven’t, and start the server process with the config. Hopefully, you will see something like this.

I0430 05:54:56.091770       1 log.go:14] Disabling "enable_last_known" feature until it is fixed... I0430 05:54:56.092200       1 log.go:14] Initializing MarketStore... I0430 05:54:56.092236       1 log.go:14] WAL Setup: initCatalog true, initWALCache true, backgroundSync true, WALBypass false: I0430 05:54:56.092340       1 log.go:14] Root Directory: /project/data/mktsdb I0430 05:54:56.097066       1 log.go:14] My WALFILE: WALFile.1525067696092950500.walfile I0430 05:54:56.097104       1 log.go:14] Found a WALFILE: WALFile.1525067686432055600.walfile, entering replay... I0430 05:54:56.100352       1 log.go:14] Beginning WAL Replay I0430 05:54:56.100725       1 log.go:14] Partial Read I0430 05:54:56.100746       1 log.go:14] Entering replay of TGData I0430 05:54:56.100762       1 log.go:14] Replay of WAL file /project/data/mktsdb/WALFile.1525067686432055600.walfile finished I0430 05:54:56.101506       1 log.go:14] Finished replay of TGData I0430 05:54:56.109380       1 plugins.go:14] InitializeTriggers I0430 05:54:56.110664       1 plugins.go:42] InitializeBgWorkers I0430 05:54:56.110742       1 log.go:14] Launching rpc data server... I0430 05:54:56.110800       1 log.go:14] Launching heartbeat service... I0430 05:54:56.110822       1 log.go:14] Enabling Query Access... I0430 05:54:56.110844       1 log.go:14] Launching tcp listener for all services...

If you see something like “Response error: Rate limit exceeded”, that’s a good sign, not a bad one, since it means the background worker successfully fetched the price data and reached to rate limit. The fetch worker will suspend for a while and restart to catch up to the current price automatically. You just need to keep it running.

Client Side

MarketStore implements JSON-RPC and MessagePack-RPC for query. MessagePack-RPC is particularly important for performance of a query on a large dataset. Thankfully, there is already python and go client library so you don’t have to implement the protocol. In this article, we use python. We start from miniconda3 image from another terminal.

$ docker run -it --rm -v $PWD/client.py:/tmp/client.py --net host continuumio/miniconda3 bash # pip install ipython pymarketstore

We have installed ipython and pymarketstore, including their dependencies. From this terminal, let’s start an ipython shell and query MarketStore data.

# ipython (base) root@hq-dev-01:/# ipython Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) Type 'copyright', 'credits' or 'license' for more information IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import pymarketstore as pymkts
In [2]: param = pymkts.Params('BTC', '1D', 'OHLCV', limit=100)
In [3]: df = pymkts.Client('http://localhost:5993/rpc').query(param).first().df()
In [4]: df[-10:] Out[4]:                               Open     High      Low    Close Volume Epoch 2018-04-14 00:00:00+00:00  7893.19  8150.00  7830.00  8003.11   9209.196953 2018-04-15 00:00:00+00:00  8003.12  8392.56  8003.11  8355.25   9739.103514 2018-04-16 00:00:00+00:00  8355.24  8398.98  7905.99  8048.93  13137.432715 2018-04-17 00:00:00+00:00  8048.92  8162.50  7822.00  7892.10  10537.460361 2018-04-18 00:00:00+00:00  7892.11  8243.99  7879.80  8152.05  10673.642535 2018-04-19 00:00:00+00:00  8152.05  8300.00  8101.47  8274.00  11788.032811 2018-04-20 00:00:00+00:00  8274.00  8932.57  8216.21  8866.27  16076.648797 2018-04-21 00:00:00+00:00  8866.27  9038.87  8610.70  8915.42  11944.464063 2018-04-22 00:00:00+00:00  8915.42  9015.00  8754.01  8795.01   7684.827002 2018-04-23 00:00:00+00:00  8795.00  8991.00  8775.10  8940.00   3685.109169

Voila! You just got the daily bitcoin price in hand in the DataFrame format. Note the second line (param = …) determines which symbol and timeframe to query, with some query predicates such as the number of rows or date range to query. From here, you can do a number of things including calculating indicators such as moving average and bollinger band, or find the statistical volume anomaly using some scipy package.

Conclusion

I want to emphasize that it is very important to build a performant historical dataset to study and develop a trading algorithm, and you can do it quickly with MarketStore as we have just walked through. This article demonstrated how to work with the bitcoin prices from GDAX, but you can hook up other data sources as well pretty easily using pymarketstore’s write method. You can also write your own custom background data fetcher.

Again, the query performance is going to be critical when in comes to backtesting, since you want to iterate quickly to get the results. now You may wonder how fast MarketStore can be. I will show the lightning fast query speed with huge data set in the next post.

In the meantime, please leave any questions in the comments or ask @AlpacaHQ regarding this tutorial. Leave your email below so we can notify you when we can grant access to the full trading platform! You can also check us out at https://alpaca.markets.

Happy algo trading!

/

MarketStore, the financial time series database, is now open source

We are happy to announce MarketStore is now open source! MarketStore is a database server optimized for financial timeseries data written in pure Go, designed and developed by Alpaca. You can think of it as an extensible DataFrame service that is accessible from anywhere in your system, at higher scalability.

Read More
/