Are You a Winner of the Alpaca Algo Trading Coding Challenge?

See Who Is Heading to the Front of the Line for Exclusive First Look Access.

Three weeks ago we threw down a challenge — solve our coding challenge and you’d go to the front of the line for those waiting to get the first look at our commission free algo trading platform.

coding.gif

Here’s the challenge we issued on May 3.

Medium - Puzzle Snippet.png

Many tried, but few succeeded in solving this puzzle. In fact, fewer than half (39 percent) of those who submitted answers were able to get the correct answer. Others found their way to the front of the line by sharing the coding challenge with friends.

Today we’re releasing the emails of those who solved the challenge and are sitting on their royal thrones right at the front of our waitlist.

See all the winners here.

 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —  

If you gave it a shot, but didn’t quite get the answer right, here’s the way the puzzle was solved:

Overall it looks like a python code snippet but also includes some pseudo-math type of expressions. Let’s start by taking apart this puzzle line by line.

Line 1 

This line talks about a URL to the JSON file. If you open this URL with your browser, it should start like this.

{“date”: [“2015–07–10”, “2015–07–11”, “2015–07–12”, “2015–07–13”, “2015–07–14”, “2015–07–15”, “2015–07–16”, “2015–07–17”, “2015–07–18”, “2015–07–19”, “2015–07–20”, “2015–07–21”, “2015–07–22”, “2015–07–23”, “2015–07–24”, “2015–07–25”, “2015–07–26”, “2015–07–27”, “2015–07–28”, “2015–07–29”, “2015–07–30”, “2015–07–31”, “2015–08–01”, “2015–08–02”, “2015–08–03”, “2015–08–04”, “2015–08–05”, “2015–08–06”, “2015–08–07”, “2015–08–08”, “2015–08–09”, “2015–08–10”, “2015–08–11”, “2015–08–12”, “2015–08…

Yes, it is a JSON with keys of “date”, “open”, “high”, “low”, “close” and “volume”. Looks like some kind of historical price data. Makes sense. Alpaca is about algo trading.

Line 2 

This line was apparently a bit tricky. The quiz meant to be as simple as literal python code to slice the string variable “file” and get “-100” as an integer but some people thought it was meant to slice the content of JSON file.

Line 3 

MA here is the Moving Average function, which is a basic indicator derived from some historical price data, and typically it takes the input data and the time period of average as the parameters.

Line 4 

Calculates MA but with unknown X, and X is the set of number that satisfies this condition.

Line 5 

What is the minimum value in X?

In short, the puzzle is asking:

Using this historical price data (line 1), let m be the 5-day moving average of open prices on 100 days (line 2) before 04/01/2018 (line 3). Find the minimal (line 5) day moving average of close prices on the same day whose absolute difference from m is bigger than the fraction of square root 2 (line 4)?

If you use python and pandas, you can find the answer in a few tens of line of code with brute-force loop and get the answer of 61. This is the correct answer.

Now that you have the answer and took a look at the JSON file content, here is another short, fun quiz. Can you tell what kind of data it is? We’ll let you know the answer in the next week update, so please do not forget to be on the email list if you haven’t already!

 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

It’s not too late to get your own spot in line to get an exclusive first look at Alpaca. Just go to https://alpaca.markets to enter your email and we’ll make sure you get in before the stampede!

 

 

/

So You Want to Trade Crypto  -  Volatility (Part 2)

The Cryptocurrency market is extremely new and has yet to mature with greater regulation and institutional involvement, leading to sharp moves and high volatility as the market grows. This is a double edged sword — heaven for day traders looking for a bit of excitement while adding significant risk to those wishing to trade longer term.

Historical Volatility

Historical Volatility comparison of asset classes.

  • GVZVIX — CBOE Volatility index for Gold
  • VIX — CBOE Volatility index for the S&P 500
  • EUVIX — CBOE Volatility index for Euro/USD pair
  • BTCVIX — Bitcoin 30 day historical volatility (Bitcoin volatility on right axis as it’s so much higher)
Volatility_comparison.png

As we can see, there is a clear correlation between the volatility of more traditional traded assets, especially so during events such as the 2009 bounce. Bitcoin, on the other hand, seems to hold limited correlation to any of these other classes of products, even when factoring out the huge difference in average volatility.

Even the volatility of equities, currently at 13.4% from the VIX, is completely dwarfed by that of Bitcoin, coming in at 70% (down from 150% earlier in the year) — and Bitcoin is historically one of the less volatile Cryptocurrencies. Cryptocurrency markets represent a huge shift in gear, incurring more risk while giving the possibility of greater reward.

How to manage the volatility

Managing downside risk is an important part of any successful trading strategy, more so in the extreme volatility of Cryptocurrency. I personally prefer to define a hard stop-loss level prior to each trade, however a “soft” soft-loss/ exit parameter can be sufficient depending on the frequency of the trading strategy.

Very few Crypto exchanges cater to higher frequency trading at the moment, making faster trading strategies more difficult if not impossible in many cases. High frequency market making or scalping strategies are less concerned with hard stop-losses, due to the extremely short exposure to directional risk.

As hold duration for each trade moves into the span of minutes, absolute risk is still relatively limited in most cases. However, as in any market, there are occasional spikes in volatility — but these can be amplified too.

 12th Apr 2018 — A move of $1200 — greater than the entire span of previous 2 weeks

12th Apr 2018 — A move of $1200 — greater than the entire span of previous 2 weeks

The risk:reward ratio of each trade is an important factor to a profitable trading system, along with the success rate 

If there is no plan for risk management, a black-swan event (such as the margin cascade seen above) could undo a week’s profits — or even wipe out a while account.

Moving into longer term strategies, such as swing trading

Volatility makes common strategies trickier. The simplest way to reduce the risk on a trade is to reduce position sizing on signals with lower confidence. For example, in forex trading, leverage is widely used to make efficient use of capital on relatively smaller movements. Meanwhile, it’d be madness to take that level of leverage on a Crypto swing trade. Risk doesn’t have to be any higher between asset classes, as long as volatility is adjusted for.

From active trading strategies into indexing, the Cryptocurrency market has provided huge returns over the past couple of years, with the total market capitalization rising from $18.3B at the start of 2017 to $613B by the start of 2018 — over 3000% growth. Unfortunately, past performance isn’t indicative of future returns — it would be impossible to continue at such a rate. Between Jan 2018 and Apr 2018, there was a pullback of 60% from which we’re still recovering.

An automatically re-balanced Cryptocurrency index fund can offer a simple but effective way to gain diversified exposure to the market, but be warned that you will have to be able to stomach the possibility of high drawdowns.

Volatility and Speculative Assets

As I mentioned in the first post of the series, Cryptocurrency cannot be analyzed in the same manner as traditional assets. You have no physical item or stake in a company to which the price is tied. The whole valuation of your token is based on the expectation of future demand, and is thus very susceptible to changes in sentiment.

A stock will be priced above the value of its dividends and assets to account for the expected growth in the business over time. However, during a market downturn, investors become less optimistic about the future as the economy constricts, driving down prices. The business itself may lose some revenue as consumers of the economy have less cash to spend, but there is still a functioning business with value.

Meanwhile, the whole success of Cryptocurrency relies on the ability of a project to attract businesses and developers to use the platform. Speculators are betting that the project will be successful in this goal long term, but there is no tangible asset to retain value if the project falters or market sentiment shifts against it.

This is one of the main reasons why the Cryptocurrency market remains so volatile compared to any other asset class

While this effect will reduce over time, as successful applications and businesses form in the ecosystem, it will not be a quick or easy process.

by Matthew Tweed

/

So You Want to Trade Crypto  - Fundamental Analysis (Part 1)

This will be a mini-series looking into: 

  • the nature of Cryptocurrency trading and investment, 
  • the key factors which shift the markets, and
  • the important differences between Crypto and more traditional traded assets.

Fundamental factors of Cryptocurrencies vs traditional assets.

In terms of investing and long term trading, Cryptocurrency can be analyzed somewhere between a stock and a commodity.

It should be seen very much as a speculative investment — in the case of most Cryptocurrencies, you’re buying into a token which represents no stake in a company nor any voting rights, similar to a commodity. However, you’re not investing into a physical asset either. You’re simply betting that over time the use-case and thus demand for your token will grow, driving up the price.

Since price over time will be relative to the success of product, it can be analyzed similarly to an investment in a start-up. The main focus when looking at a token should be the team and their ability to deliver a working product that will actually be used. Not only this, you should also take
into consideration the use of the token itself within their product.

Will growth of the product actually lead to higher demand for the token, or is it simply tacked on as an after thought?

Additionally, Cryptocurrency projects seem to have a very different valuation split to previous technologies. A shift in valuation from the applications built on top of a protocol to the Blockchain protocol itself. No-one is investing in creating a new protocol layer for the internet, as the captured value comes from the applications and companies build upon it, such as Google, Facebook and Amazon.

On the other hand, Cryptocurrencies such as Bitcoin and Ethereum have market caps of $100B while very few applications have even been produced yet, let alone applications in wide use outside of the sphere of Crypto-enthusiasts.

This may be a symptom of a wider speculative bubble around the Cryptocurrency market as a whole, or simply a fact of the technology — requiring a sufficient valuation to be maintained to allow efficient flow of payments. That being said, we haven’t seen a truly successful product built
on Blockchain yet, maybe the Blockchain killer app will dwarf current valuations in the long term - only time will tell.

What to look for in projects

The core proposition of Cryptocurrencies over the traditional financial system is the fact that you don’t have to trust a third party. 

Trust-less decentralization

A transaction can be sent to anyone anywhere in the world without needing a bank or escrow account in the middle. This property of trust-less decentralization should be the backbone of any good Cryptocurrency.

However, simple transactions are a very basic and limited use-case, which has been offered by Bitcoin since its first release in 2009. A project has to offer a unique and useful advancement on basic Blockchain technology. Many new Cryptos are becoming “smart contract platforms”.

Smart contracts are pieces of code which can be run on top of the Blockchain, in the same decentralized and trust-less manner as payments. This allows the easy creation of full applications on top of the basic transaction layer, providing much wider uses for businesses and developers.

Longterm Scalability

Longer term, one of the fundamental issues that needs to be solved in Blockchain technology is that of scaling. Since the basic Blockchain model relies on all connected nodes to process every transaction, it quickly runs into limitations of how fast a payment can be distributed and verified, not to mention that of storage.

While there are many projects out there which claim “1,000,000 transactions per second”, almost all of these approaches are either impractical in real life or break the fundamental premise of a Cryptocurrency being trust-less and decentralized. Its relatively easy to achieve “1M tps” when running high powered servers in the same location that all have implicit trust.
However, in reality, flow of transactions is limited by latency between nodes around the globe and the fact that we shouldn’t rely on a central node for verification — otherwise why don’t we just use a regular bank?

The scaling issue is being progressed on with such technologies as “sharding” or a change from a traditional Blockchain to a “DAG”. In the long term, these are the kind of advancements which will achieve reliable scaling, rather than misleading promises from certain projects.

Product Value w/ Blockchain

Finally, and most importantly, you should consider who the Cryptocurrency is developed for and whether it is truly a useful product to them. Many have tried applying Blockchain in a misguided manner to any and all systems to join in with the hype. However, unless you need the very specific properties of a decentralized and trust-less system, Blockchain is no more than a glorified
database.

Final thoughts

While there is innovation left and right in the Cryptocurrency space, caution is still strongly advised. Out of the over 1500 coins and tokens in existence, only a handful of projects are likely to survive. While the vast majority will fail in the long run, due diligence pays off — helping us sift through the rubbish to find those which have a chance at leading the revolution of decentralized finance.

by Matthew Tweed

/

50x Faster Bitcoin Price Data Powered by MarketStore for AI Trading

In our last post “How to Setup Bitcoin Historical Price Data for Algo Trading in Five Minutes”, we introduced how to set up bitcoin price data in five minutes and we got a lot of good feedback and contributions to the open source MarketStore.

 Photo by  chuttersnap  on  Unsplash

Photo by chuttersnap on Unsplash

The data speed is really important

Today, I wanted to tell you how fast MarketStore is using the same data so that you can see the performance benefit of using the awesome open source financial timeseries database.

Faster data means more backtesting and more training in machine learning

Faster data means more backtesting and more training in machine learning for our trading algorithm.  We are seeing a number of successful machine learning-based trading algos in the space, but one of the key points we learned is the data speed is really importantIt's important not just for backtesting, but also for training AI-style algorithms since it by nature requires an iterative process.

This is another post to walk you through step by step. But TL;DR, it is really fast.

Setup

structure.001.jpeg

Last time, we showed how to setup the historical daily bitcoin price data with MarketStore.

This time, we store all the minute-level historical prices using the same mechanism called background worker, but with a slightly different configuration.

 

root_directory: /project/data/mktsdb
listen_port: 5993
# timezone: "America/New_York"
log_level: info
queryable: true
stop_grace_period: 0
wal_rotate_interval: 5
enable_add: true
enable_remove: false
enable_last_known: false
triggers:
 - module: ondiskagg.so
   on: "*/1Min/OHLCV"
   config:
     destinations:
       - 5Min
       - 15Min
       - 1H
       - 1D
bgworkers:
 - module: gdaxfeeder.so
   name: GdaxFetcher
   config:
     query_start: "2016-01-01 00:00"
     base_timefame: “1Min”
     symbols:
       - BTC

Almost 2.5 years with more than 1 million bars

The difference from last time is that background worker is configured to fetch 1-minute bar data instead of 1-day bar data, starting from 2016-01-01.  That is almost 2.5 years with more than 1 million bars. You will need to keep the server up and running for a day or so to fill all the data, since GDAX’s historical price API does not allow you to fetch that many data points quickly.

Again, the data fetch worker carefully controls the data fetch speed in case the API returns “Rate Limit” error. So you just need to sleep on it.

Additional configuration here is something called “on-disk aggregate” trigger.  What it does is to aggregate 1-minute bar data for lower resolutions (here 5 minutes, 15 minutes, 1 hour, and 1 day).

Check the longer time horizon to verify the entry/exit signals

In a typical trading strategy, you will need to check the longer time horizon to verify the entry/exit signals even if you are working on the minute level. So it is a pretty important feature. You would need pretty complicated LEFT JOIN query to achieve the same time-windowed aggregate in SQL. But with MarketStore, all you need is this small section in the configuration file.

The machine we are using for this test is a typical Ubuntu virtual machine with 8 of Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz, 32GB RAM and SSD.

The Benchmark

Unfortunately lots of people in this space are using some sort of SQL database

We are going to have a DataFrame object in python which holds all the minute level historical price data of bitcoin since January of 2016 from the server.  We compare MarketStore and PostgreSQL.

PostgreSQL is not really meant to be the data store for this type of data, but unfortunately lots of people in this space are using some sort of SQL database for this purpose since there is no other alternative.  That’s why we built MarketStore.

The table definition of the bitcoin data in PostgreSQL side looks like this.

btc=# \d prices
              Table "public.prices"
 Column |            Type             | Modifiers
--------+-----------------------------+-----------
 t      | timestamp without time zone |
 open   | double precision            |
 high   | double precision            |
 low    | double precision            |
 close  | double precision            |
 volume | double precision            |

The code looks like this.

# For postgres
def get_df_from_pg_one(conn, symbol):
    tbl = f'"{symbol}"'
    cur = conn.cursor()
    # order by timestamp, so the client doesn’t have to do it
    cur.execute(f"SELECT t, open, high, low, close, volume FROM {tbl} ORDER BY t")
    times = []
    opens = []
    highs = []
    lows = []
    closes = []
    volumes = []
    for t, open, high, low, close, volume in cur.fetchall():
        times.append(t)
        opens.append(open)
        highs.append(high)
        lows.append(low)
        closes.append(close)
        volumes.append(volume)

    return pd.DataFrame(dict(
        open=opens,
        high=highs,
        low=lows,
        close=closes,
        volume=volumes,
    ), index=times)

# For MarketStore
def get_df_from_mkts_one(symbol):
    params = pymkts.Params(symbol, '1Min', 'OHLCV')
    return pymkts.Client('http://localhost:6000/rpc'
                         ).query(params).first().df()

You don’t need much client code to get the DataFrame object

The input and output is basically the same, in that one symbol name is given, query the remote server over the network, and get one DataFrame.  One strong benefit of MarketStore is you don’t need much client code to get the DataFrame object since the wire protocol is designed to give an array of numbers efficiently.

The Result

First, PostgreSQL

%time df = example.get_df_from_pg_one(conn, 'prices')
CPU times: user 8.11 s, sys: 414 ms, total: 8.53 s
Wall time: 15.3 s

And MarketStore

%time df = example.get_df_from_mkts_one('BTC') 
CPU times: user 109 ms, sys: 69.5 ms, total: 192 ms Wall time: 291 ms 

Both results of course look the same like below.

In [21]: df.head()
Out[21]:
                       open    high     low   close   volume
2016-01-01 00:00:00  430.35  430.39  430.35  430.39   0.0727
2016-01-01 00:01:00  430.38  430.40  430.38  430.40   0.9478
2016-01-01 00:02:00  430.40  430.40  430.40  430.40   1.6334
2016-01-01 00:03:00  430.39  430.39  430.36  430.36  12.5663
2016-01-01 00:04:00  430.39  430.39  430.39  430.39   1.9530

 

50 times difference

A bitcoin was about $430 back then… Anyway, you can see the difference between 0.3 vs 15 seconds which is about 50 times difference. Remember, you may need to get the same data again and again for different kinds of backtesting and optimization as well as ML training.

Also you may want to query not just bitcoins but also other coins, stocks and fiat currencies, since the entire database wouldn’t fit into your main memory usually.

Scalability advantage in MarketStore

MarketStore can serve multiple symbol/timeframe in one query pretty efficiently, whereas with PostgreSQL and other relational databases you will need to query one table at a time, so there is also scalability advantage in MarketStore when you need multiple instruments.

Querying 7.7K symbols for US stocks

To give some sense of this power, here is the result of querying 7.7K symbols for US stocks done as an internal testing.

%time dfs = example.get_dfs_from_pg(symbols) 
CPU times: user 52.9 s, sys: 2.33 s, total: 55.3 s Wall time: 1min 26s 
%time dfs = example.get_dfs_from_mkts(symbols) 
CPU times: user 814 ms, sys: 313 ms, total: 1.13 s Wall time: 6.24 s

Again, the amount of data is the same, and in this case each DataFrame is not as large as the bitcoin case, yet the difference to expand to large number of instruments is significant (more than 10 times).  You can imagine in real life these two (per instrument and multi-instruments) factors multiply the data cost.

Alpaca has been using MarkStore in our production

Alpaca has been using MarkStore in our production for algo trading use cases both in our proprietary customers and our own purposes.  It is actually amazing that this software is available to everyone for free, and we leverage this technology to help our algo trading customers (early access signup is here).

Thanks for reading and enjoy algo trading!


 

/

Algo Trading Daily News Scan by Alpaca - May 17th 2018

The 25 Highest-Earning Hedge Fund Managers And Traders

(www.forbes.com)

In total, the 25 highest-earning hedge fund managers and traders made a combined $16.8 billion in 2017, making it the best year for the top hedge fund managers since 2013. Forbes includes in its analysis hedge fund managers and traders who now mostly or even exclusively manage their own money and some of them can be found in the top 10 highest earners of 2017.

 

A Quantitative System For Trading Netflix Stock

(seekingalpha.com)

Knowing that the stock did so well over time, you may feel inclined to believe that maybe holding Netflix stock was not such a big challenge. Nevertheless, the stock lost more than 70% of its value in three different years: 2002, 2004, and 2011. In addition to that, Netflix had two more years with drawdowns of more than 50% in 2008 and 2012.

 

Simple way to come up with trading strategies using order flow data

(alpaca.markets)

We can see that the average next-day returns are almost monotonically increasing with the previous day’s order flow. The spread between the top and bottom quintiles is about 12 bps, which annualizes to about 30%. This doesn’t mean we will generate 30% annualized returns, since we won’t be able to arbitrage between today’s return (if the order flow is in the top or bottom quintile) with some previous day’s return when its order flow was in the opposite extreme. Nevertheless, it is encouraging.

 

Algorithmic Trading and TCA Adoption Driven by FX Global Code

(www.tradersmagazine.com)

Trading electronically in FX has been commonplace for years through Request-For-Quote (RFQ) or click trading. However, the increasing fragmentation of liquidity, accelerating speed of ECN matching engines and market participants, coupled with the proliferation of new trading tools, has created a whole host of fresh challenges and opportunities for corporate treasury departments.

 

Computers drive trading in coffee and cocoa markets

(www.ft.com)

Specialised markets such as cocoa and coffee are becoming the next frontier for the waves of buying and selling generated by algorithmic trading, which has long since carved out a presence in currencies, equities and interest rates.

/

Simple way to come up with trading strategies using order flow data

FX Order Flow as a Predictor

By Ernest P. Chan, Managing Member of QTS Capital Management, LLC.

Order flow is signed trade size, and it has long been known to be predictive of future price changes. (See Lyons, 2001, or Chan, 2017.) The problem, however, is that it is often quite difficult or expensive to obtain such data, whether historical or live. This is especially true for foreign exchange transactions which occur over-the-counter. Recognizing the profit potential of such data, most FX market operators guard them as their crown jewels, never to be revealed to customers. But recently FXCM, a FX broker, has kindly provided me with their proprietary data, and I have made use of that to test a simple trading strategy using order flow on EURUSD.

First, let us examine some general characteristics of the data. 

It captures all trades transacted on FXCM occurring in 2017, time stamped in milliseconds, and with their trade prices and signed trade sizes. The sign of a trade is positive if it is the result of a buy market order, and negative if it is the result of a sell. If we take the absolute value of these trade sizes and sum them over hourly intervals, we obtain the usual hourly volumes (click to enlarge) aggregated over the 1 year data set:

image-1.png

The highest volume occurs between 15:00–16:00 London time.

Next, I compute the daily total order flow of EURUSD… 

(with the end of day at New York’s midnight), and I establish a histogram of the last 20 days’ daily order flow. I then determine the average next-day return of each daily order flow quintile. (I.e. I bin a next-day return based on which quintile the prior day’s order flow fell into, and then take the average of the returns in each bin.) The result is satisfying:

img-epchan2.png

We can see that the average next-day returns are almost monotonically increasing with the previous day’s order flow. The spread between the top and bottom quintiles is about 12 bps, which annualizes to about 30%. This doesn’t mean we will generate 30% annualized returns, since we won’t be able to arbitrage between today’s return (if the order flow is in the top or bottom quintile) with some previous day’s return when its order flow was in the opposite extreme. Nevertheless, it is encouraging. Also, this is an illustration that even though order flow must be computed on a tick-by-tick basis (I am not a fan of the bulk volume classification technique), it can be used in low-frequency trading strategies.

One may be tempted to also regress future returns against past order flows, but the result is statistically insignificant. Apparently only the top and bottom quintiles of order flow are predictive. This situation is actually quite common in finance, which is why linear regression isn’t used more often in trading strategies.

Finally, one more sanity check before backtesting. 

I want to see if the buy trades (trades resulting from buy market orders) are filled above the bid price, and the sell trades are filled below the ask price. Here is the plot for one day (times are in New York):

img-epchan3.png

We can see that by and large, the relationship between trade and quote prices is satisfied. We can’t really expect that this relationship holds 100%, due to rare occasions that the quote has moved in the sub-millisecond after the trade occurred and the change is reported as synchronous with the trade, or when there is a delay in the reporting of either a trade or a quote change.

So now we are ready to construct a simple trading strategy that uses order flow as a predictor. 

We can simply

  • buy EURUSD at the end of day when the daily flow is in the top quintile among its last 20 days’ values, 
  • and hold for one day, 
  • and short it when it is in the bottom quintile. 

Since our daily flow was measured at midnight New York time, we also define the end of day at that time. (Similar results are obtained if we use London or Zurich’s midnight, which suggests we can stagger our positions.) In my backtest, I have subtracted 0.20 bps commissions (based on Interactive Brokers), and I assume I buy at the ask and sell at the bid using market orders. The equity curve is shown below:

img-epchan4.png

The CAGR is 13.7%, with a Sharpe ratio of 1.6. Not bad for a single factor model!

Acknowledgement: I thank Zachary David for his review and comments on an earlier draft of this post, and of course FXCM for providing their data for this research.

This post originally appeared on Quantitative Trading in February.


Those who signup for Alpaca will be entered in a lucky drawing to receive a free copy of “Machine Trading,” by Author Ernie Chan.

Medium - Machine Trading.png

About the Author: Ernie is the Managing Member of QTS Capital Management, LLC., a commodity pool operator and trading advisor. Ernie has worked for various investment banks (Morgan Stanley, Credit Suisse, Maple) and hedge funds (Mapleridge, Millennium Partners, MANE) since 1997. He received his Ph.D. in physics from Cornell University and was a member of IBM’s Human Language Technologies group before joining the financial industry. He is the author of “Quantitative Trading: How to Build Your Own Algorithmic Trading Business”, “Algorithmic Trading: Winning Strategies and Their Rationale”, and “Machine Trading: Deploying Computer Algorithms to Conquer the Markets”. Find out more about Ernie at www.epchan.com.

===

 

/

Algo History Made This Weekend

We’re very happy to report we have helped make a little financial history this weekend when we admitted the very first users from our waitlist to the Alpaca platform and for the first time ever algo trades were conducted without a incurring a single penny of trading commissions! These users 

  1. ported their algos directly on to our platform, 
  2. deposited funds into their account and 
  3. launched their algo trading strategies 

all on our groundbreaking commission free platform. Certainly, this is an exciting milestone for our startup, but it’s a huge step forward for algo trading as a whole.

Automated trading is the wave of the future on Wall Street. 

Last year, J.P. Morgan told CNBC that only 10 percent of stock trades today on U.S. exchanges are actually ordered by human traders.

But it’s also the wave of the future across America. More than ever, there are qualified individuals who can write algorithms and use them to trade and do it as well as any institution. We think it’s critical that there is a platform that lowers the cost of algo trading to encourage and inspire this revolutionary change in consumer trading.

It seems like we cannot count on U.S. regulators to do much to force markets to respond to this shift to automated trading. India, the E.U., even the post EU United Kingdom seems to be doing more than the U.S. when it comes to making algo trading more affordable and less technologically challenging. Free market solutions like our Alpaca are in fact the real driving force in eliminating trading commissions on algo trades.

So please join us on this journey to make algo trading commission free for everyone. And it won’t be long before the algo trading wave forces the truly transformational end to trading commissions everywhere. Push this movement forward and sign up for commission free algo trading!

/

Algo Trading Daily News Scan by Alpaca - May 14th 2018

Algo Trading Daily News Scan by Alpaca - May 14th, 2018

The Top Algo Trading News Stories and Headlines From Around the Web

Algorithmic Trading and TCA Adoption Driven by FX Global Code

(www.tradersmagazine.com)

“Today the speed of trading, coupled with the number of venues and increasing variety of order types, make it impossible for a human trader to replicate algorithmic behaviour on one order, let alone if the trader has multiple orders to trade simultaneously.”

 

Algo Traders Warn U.K. Over MiFID II Rules

(www.bloomberg.com)

“Financial firms are pressing the U.K. to stick as closely as possible to the European Union’s MiFID II restrictions on algorithmic trading, warning the Bank of England against imposing even tougher rules on the industry ahead of Brexit.”

 

HOW AI IS CHANGING THE FACE OF FINANCE

(www.i4u.com)

“It is a little like a meteorologist looking at two weather systems and seeking to predict how they will change over time, and how they might interact with one another. To stretch the analogy, when you try to decide on the best currency pair for Forex trading, you are selecting two out of more than 100 different weather systems to pitch against one another.”

 

Why Algorithmic Trading Really Works

(www.moneymorning.com.au)

“People often ask how I got my start. They typically want to know how I became a ‘system trader’ — it’s not the sort of job you hear about at a school careers night.”

 

How to become an algo trader

(news.efinancialcareers.com)

“Knowing the data structures, algorithms and C++ or another programming language inside and out is critical. That attribute is not easily available, and people really value that. Most [financial services and fintech] firms will hire a great programmer without trading experience over a mediocre programmer who knows trading.”

 

Automated Trading Broadens Accessibility to Crypto

(www.investopedia.com)

“While traditional asset markets like stocks have set trading hours and no weekend trading, cryptocurrency prices move around the clock, every day of the year. For most investors, staying alert and monitoring an investment portfolio around the clock is not just unfeasible, but also impractical. While automated trading has been around for years, it was largely inaccessible for most ordinary traders, remaining in the purview of sophisticated investors until this point. However, in keeping with the democratization of finance instilled by cryptocurrency, algorithmic trading is being simplified for mass-market consumption.”

Footer_Disclosure.png
/

So commission-free trading is pretty legit now right?!

The big news today in stock trading is the $365 million raise by Robinhood that increases the company’s valuation to more than $5 billion. In less than five years, Robinhood’s commission free trading platform has attracted more than 4 million users, managed more than $150 billion in transactions and saved users more than $1 billion in fees.

What this explosive growth from Robinhood really means is that the days of charging commissions on stock trades is quickly going to become a relic of the past. And while Robinhood’s success is a milepost on the route to the end of commissions, we believe it is actually the rapid growth of algo trading that will deal the death blow to trading commissions.

Medium - commission free.png

Today, we are seeing unprecedented growth in algo trading on the institutional and, more importantly, the individual level. 

  • Financial regulators in India reported last month that algo trading has grown at least 50 percent in less than a decade. 
  • Bloomberg reported last week that automated trades now constitute about 80 percent of activity on U.S. exchanges.

The increase in the volume of algo trading must correlate to an increase in frequency of trading. And this surge in the frequency of trading can be limited by out-of-date commissions on individual trades. Commissions are becoming an impediment to trading progress because they conform to a method of trading that has already faded into the past.

The private sector is already responding to this need and demand for commission free trading. Robinhood is the most well known example. Our new startup Alpaca will soon allow commission free algo trading.

Soon we believe financial regulators will need to get involved and require or regulate trading in a way that more appropriately accommodates automated and algo trading.

Let’s return to India again. 

Last month, the country’s leading financial regulator, Sebi, announced it was seeking reforms on leading financial exchanges that would support algo trading. Sebi says it may ask exchanges to offer some services for free to algo traders, including a data fee for testing algos and reduced costs for colocation facilities within the exchanges themselves.

And in London…

Algo traders are pressing the UK to stick closely to the European Union standards on algo trading as the country backs out of Brexit. New financial rules that have circulated unofficially in London show that UK regulator may favor a requirement that any changes to algo trading strategies be approved by regulators before being implemented, a burden algo traders have branded as ridiculous.

At any rate, it is clear that around the globe algo trading is reshaping the rules, the fees and the markets

And it won’t be long before the algo trading wave forces the truly transformational end to trading commissions everywhere. Push this movement forward and sign up for commission free algo trading!

Footer_Disclosure
/

How to Setup Bitcoin Historical Price Data for Algo Trading in Five Minutes

Alpaca platform is now accepting signups for our algo trading waitlist and we do believe trading automation is the future. 

 Photo by  Chris Liverani  on  Unsplash
One of the great benefits of algorithmic trading is that you can test your trading strategy against historical data. 

Especially for new strategies you developed on your own, you don’t really know how it will perform without testing it against reliable data. Algo trading is not that different from software development. Today, most good software code is continuously tested. As the code evolves, developers continually test it against real use cases to make sure that alterations won’t result in future failures. You want to test your algo when you drop the very first version. But you also want to test the algo when you make changes or adjustments while the algo is running.

Here is one problem you face; data. 

While it is always good to test as many angles as possible, running a number of backtests is going to be a very data-intensive workload that requires access to enough data to have visibility into a very long history. This is why you cannot run this iteration using GDAX API directly. You need to store the data somewhere for your own purposes.

The Tutorial

So the first question to come to mind is always how to get the data and prepare it for successful backtesting. Well, today I am going to tell you how to use MarketStore to acquire a long history of Bitcoin price data for this purpose of running the most accurate backtest possible. And this setup tutorial is going to be quick. Don’t waste your time setting up this and that. All you need is:

  • docker (either on Windows, Mac or Linux),
  • a console terminal,
  • and a big cup of strong coffee! ☕

Got’em? Alright, let’s get started.

 

Architecture

Medium - bitcoin data1.png

Here is the high level picture of today’s system. We will start a MarketStore instance using docker container, and run a background worker that calls GDAX price API so that we can pull the bitcoin historical price from their endpoint quickly and make it available for backtest clients to query over HTTP.

We will start another container for the client using python anaconda with python3 image. We use the official client package named pymarkestore. You will get a DataFrame from MarketStore.

Setup MarkeStore Server

There is the official build of MarketStore docker image today publicly available in DockerHub, but first, let’s write a config file for the server.

In the github repository you can find an example config file in YAML format: https://github.com/alpacahq/marketstore/blob/master/mkts.yml but I’m putting our example here.

root_directory: /project/data/mktsdb
  listen_port: 5993
  log_level: info
  queryable: true
  stop_grace_period: 0
  wal_rotate_interval: 5
  enable_add: true
  enable_remove: false
  enable_last_known: false
  bgworkers:
    - module: gdaxfeeder.so
      name: GdaxFetcher
      config:
        symbols:
         - BTC 
        base_timeframe: "1D" 
        query_start: "2018-01-01 00:00"

This configures the server so that it fetches the GDAX historical price API for 1-day bars since 2018–01–01. Save this config as $PWD/mkts.yml file. The server listens on the port 5993 as default. Now let’s bring up the server.

$ docker run -v $PWD/mktsdb:/project/data/mktsdb -v $PWD/mkts.yml:/tmp/mkts.yml --net host alpacamarkets/marketstore:v2.1.1 marketstore -config /tmp/mkts.yml

The server should automatically download the docker images from DockerHub if you haven’t, and start the server process with the config. Hopefully, you will see something like this.

I0430 05:54:56.091770       1 log.go:14] Disabling "enable_last_known" feature until it is fixed...
I0430 05:54:56.092200       1 log.go:14] Initializing MarketStore...
I0430 05:54:56.092236       1 log.go:14] WAL Setup: initCatalog true, initWALCache true, backgroundSync true, WALBypass false:
I0430 05:54:56.092340       1 log.go:14] Root Directory: /project/data/mktsdb
I0430 05:54:56.097066       1 log.go:14] My WALFILE: WALFile.1525067696092950500.walfile
I0430 05:54:56.097104       1 log.go:14] Found a WALFILE: WALFile.1525067686432055600.walfile, entering replay...
I0430 05:54:56.100352       1 log.go:14] Beginning WAL Replay
I0430 05:54:56.100725       1 log.go:14] Partial Read
I0430 05:54:56.100746       1 log.go:14] Entering replay of TGData
I0430 05:54:56.100762       1 log.go:14] Replay of WAL file /project/data/mktsdb/WALFile.1525067686432055600.walfile finished
I0430 05:54:56.101506       1 log.go:14] Finished replay of TGData
I0430 05:54:56.109380       1 plugins.go:14] InitializeTriggers
I0430 05:54:56.110664       1 plugins.go:42] InitializeBgWorkers
I0430 05:54:56.110742       1 log.go:14] Launching rpc data server...
I0430 05:54:56.110800       1 log.go:14] Launching heartbeat service...
I0430 05:54:56.110822       1 log.go:14] Enabling Query Access...
I0430 05:54:56.110844       1 log.go:14] Launching tcp listener for all services...

If you see something like “Response error: Rate limit exceeded”, that’s a good sign, not a bad one, since it means the background worker successfully fetched the price data and reached to rate limit. The fetch worker will suspend for a while and restart to catch up to the current price automatically. You just need to keep it running.

Client Side

MarketStore implements JSON-RPC and MessagePack-RPC for query. MessagePack-RPC is particularly important for performance of a query on a large dataset. Thankfully, there is already python and go client library so you don’t have to implement the protocol. In this article, we use python. We start from miniconda3 image from another terminal.

$ docker run -it --rm -v $PWD/client.py:/tmp/client.py --net host continuumio/miniconda3 bash
# pip install ipython pymarketstore

We have installed ipython and pymarketstore, including their dependencies. From this terminal, let’s start an ipython shell and query MarketStore data.

# ipython
(base) root@hq-dev-01:/# ipython
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import pymarketstore as pymkts
In [2]: param = pymkts.Params('BTC', '1D', 'OHLCV', limit=100)
In [3]: df = pymkts.Client('http://localhost:5993/rpc').query(param).first().df()
In [4]: df[-10:]
Out[4]:
                              Open     High      Low    Close Volume
Epoch
2018-04-14 00:00:00+00:00  7893.19  8150.00  7830.00  8003.11   9209.196953
2018-04-15 00:00:00+00:00  8003.12  8392.56  8003.11  8355.25   9739.103514
2018-04-16 00:00:00+00:00  8355.24  8398.98  7905.99  8048.93  13137.432715
2018-04-17 00:00:00+00:00  8048.92  8162.50  7822.00  7892.10  10537.460361
2018-04-18 00:00:00+00:00  7892.11  8243.99  7879.80  8152.05  10673.642535
2018-04-19 00:00:00+00:00  8152.05  8300.00  8101.47  8274.00  11788.032811
2018-04-20 00:00:00+00:00  8274.00  8932.57  8216.21  8866.27  16076.648797
2018-04-21 00:00:00+00:00  8866.27  9038.87  8610.70  8915.42  11944.464063
2018-04-22 00:00:00+00:00  8915.42  9015.00  8754.01  8795.01   7684.827002
2018-04-23 00:00:00+00:00  8795.00  8991.00  8775.10  8940.00   3685.109169

Voila! You just got the daily bitcoin price in hand in the DataFrame format. Note the second line (param = …) determines which symbol and timeframe to query, with some query predicates such as the number of rows or date range to query. From here, you can do a number of things including calculating indicators such as moving average and bollinger band, or find the statistical volume anomaly using some scipy package.

Conclusion

I want to emphasize that it is very important to build a performant historical dataset to study and develop a trading algorithm, and you can do it quickly with MarketStore as we have just walked through. This article demonstrated how to work with the bitcoin prices from GDAX, but you can hook up other data sources as well pretty easily using pymarketstore’s write method. You can also write your own custom background data fetcher.

Again, the query performance is going to be critical when in comes to backtesting, since you want to iterate quickly to get the results. now You may wonder how fast MarketStore can be. I will show the lightning fast query speed with huge data set in the next post.

In the meantime, please leave any questions in the comments or ask @AlpacaHQ regarding this tutorial. Alpaca’s platform is accepting sign ups on our waitlist. If you haven’t signed up for the waitlist, leave your email below so we can notify you when we can grant access to the full trading platform! You can also check us out at https://alpaca.markets.

Happy algo trading!

/