Algo Trading News Headlines 7/26/2018

Look Ma, No Code! Meet The IITians Who’ve Built A Code-Free Algo-Trading Platform


“Currently, almost 42% of the trade in the Indian market is done through automated algorithmic trading systems. But most of this trading activity is done by high net worth individuals (HNIs) and AIMs. We feel that the retail guys haven’t been given enough horsepower in it. So, we are trying to do just that,” says Ayush Gangwar, CEO of Kuants.

Photo by  Tra Nguyen  on  Unsplash

Photo by Tra Nguyen on Unsplash

Data, Data, Data! 11 Great Financial Data Vendors


Whether you’re a financial firm or an individual trader, financial data is key for putting together any good strategy. With so many vendors on the market today, many good options get lost in the noise. Here are 11 great financial data vendors.

Can using Bitcoin trading robots help you earn money?


With a set of pre-written codes that help in directing the Bitcoin Trading Robots take decisions on your behalf. Moreover, one can find these robots to be more intelligent in it and thus learn from the market trends as well.

New course in algorithmic trading at Saïd


The technique works through using algorithms to replicate patterns in behaviour, and using the resulting estimates to make investment choices, removing human bias and emotion altogether.

Although algorithmic trading has been used for some time, its dominance in financial sectors is growing, and is estimated by the School to account for 20% of hedge funds.

Quant hedge funds lose their allure as performance sags


Investor inflows to computer-powered “quantitative” hedge funds have halved this year to the most sluggish pace since 2009, after a spate of poor performance from many of the industry’s biggest players.


So You Want to Trade Crypto - Hedging with Cryptocurrency and correlation structure (Part 6)

As a new asset class with historically low correlation to traditional financial products, many see Cryptocurrencies as a useful hedging tool against global downturns. However, the structure of Crypto volatility and correlation relative to market capitalization may prove somewhat detrimental to this use-case.

Photo by  Tyler Milligan  on  Unsplash

A story of Volatility

(Raw data from . These charts show the mean of the 60 day annualized volatility from 1st Jan 2017 to time of writing.)

(Raw data from These charts show the mean of the 60 day annualized volatility from 1st Jan 2017 to time of writing.)

As within equity markets, we see a small decrease in volatility as the market cap of coins increase (albeit with a relatively low correlation). This can be likened to blue-chip stocks vs mid-caps, with the former providing greater stability due to their established dominance in their respective sectors.

Although market cap is a slightly misleading metric when applied to Cryptocurrencies, it at least implies a higher value to a coin - thus requiring more money to shift its direction dramatically. That being said, volatility has been higher across the board over the last couple of years as Crypto shifted from the accumulation phase post 2013 into the major bull run. Finally pushing to record high as we moved into the final phase of the bull run and subsequent bear market as we entered 2018.

This structure of volatility allows Crypto portfolios and indexes to be constructed similarly to those of equities: high-cap only selection for reduced risk and volatility; mid-caps for higher risk and reward; or a more diversified index to try to capture a middle ground.

The Trend of Correlation

(Raw data from . These charts show the mean of the 60 day Pearson’s Correlation Coefficient against Bitcoin USD from 1st Jan 2017 to time of writing.)

(Raw data from These charts show the mean of the 60 day Pearson’s Correlation Coefficient against Bitcoin USD from 1st Jan 2017 to time of writing.)

Here we see nearly zero correlation between the market capitalization of a coin and its average correlation to Bitcoin (the historical leader of the Cryptocurrency space).

While this disproves the theory of high cap Cryptos holding a closer correlation to Bitcoin, it highlights the extremely high levels of correlation present throughout the market. This, as mentioned in previous posts, is likely due to the highly speculative and sentiment driven nature of the market, along with its relative immaturity compared to more traditional traded assets.

Interestingly, there isn’t much difference between the mean of correlation and the mean of absolute (positive only) correlation, meaning that we rarely see any negative correlation between ALT/USD pairs and BTC/USD.

Cryptocurrency as an Asset Class for hedging

Crypto holds the useful property of historically low correlation to other asset classes, such as equity and commodities, suggesting it to be a good hedge against external global factors. However, there are two main issues to this plan: Cryptocurrency has never weathered a global financial crisis; and the internal correlation within the Crypto space.

Since Bitcoin, and the rest of the Cryptocurrency market, has been experiencing its own market cycles due to its rapid growth over the past few years, any fluctuations due to correlation with equity markets has been almost unnoticeable - leading many to speculate that Cryptocurrency would continue this trend and make a good hedging tool against global downturns.

This observation happens to come on the back of a decade of huge growth in both US and global equity markets. Investors have been increasingly complacent in their gains over the past few years, and are happy to take greater and greater risks, betting money on more speculative assets such as Cryptocurrencies. However, such high yield assets are always the first to tumble at the onset of a recession, as investors scramble claw back their risk as their other positions drop.

Always "Different This Time"

Many will claim that its somehow “different this time” - it always is until the inevitable pullback. This was true of the dot-com bubble and I wouldn’t be surprised if the same fate will hold true for Cryptocurrency during a global dip. Not to say that Cryptocurrencies won’t be successful long term - the internet didn’t exactly disappear after 2000. But it should be approached with the same caution as any other high risk investment.

As alluded to in the first half of the article, the levels of volatility and correlation in Cryptocurrency make it difficult to create a well diversified portfolio - no matter what you pick you’re still at the mercy of Bitcoin and can incur the same volatility spikes and drawdowns.

While it may be possible to hedge a portfolio by shorting Bitcoin itself and creating synthetic ALT/BTC pairs, this won’t be able to eliminate the sensitivity of low-mid cap coins to shifts in market sentiment, so would have to be more actively managed.

All-in-all, Cryptocurrencies provide an interesting new opportunity for traders and investors alike - with high risk but much higher reward possibilities. They will not be a miracle financial product, nor a get rich quick scheme - but they can provide something truly new and different for those who have the time to understand and appreciate them.

By Matthew Tweed


So You Want to Trade Crypto - Exploiting Cryptocurrency Correlation (Part 5)

There is correlation within any sector or asset class, however there are particularly interesting patterns in Cryptocurrency due to the new and speculative nature of the market, along with its historical pairs structure.

Weakness and Strength

(Raw USD pairs from , raw BTC pairs from )

(Raw USD pairs from, raw BTC pairs from

Historically, the Cryptocurrency space has been dominated and led by Bitcoin, with Bitcoin’s 80% — 90% of total market cap only starting to be challenged in the last couple of years, as covered in “Market Cap Distribution and Rise of Altcoins”. This huge shift in capital distribution caused a bloom in many Altcoins during the major bull run of 2017.

However, despite this redistribution of power in the market, a correlation between different altcoins and Bitcoin stayed strong throughout 2017, suggesting that the market is still centering around Bitcoin both as an indicator of general sentiment and health and, as a safe haven asset.

During the bulk of the bull run, correlation of USD pairs stayed high, with the notable exception of periods prior to Bitcoin pullbacks, such as the dips from $3k and $5k. This seems to form a bit of a leading indicator (albeit a very noisy one), as a divergence in altcoin movement appears to precede a local top and a pullback.

Correlation and investor sentiment

The BTC pairs also tell an interesting tale, moving into negative correlation as the Bitcoin trend weakened before bouncing back once a bottom had been reached.

This view of combined market correlation can also give clues to the sentiment of investors. During bull markets, projects have a high-value premium based on the expectation of future success, meaning that while correlation stays generally positive, the price movement of projects will shift around based on their own news and merits — lowering overall correlation.

However, if we look to 2018 and the bearish trend, we see a very different pattern as fear enters the market. Correlation tends towards 1.0 in a bear market, as sell-offs are sharp across the board due to the cycle of panic. Individual projects are no longer values on their own merits, instead being sold off at whatever price they’ll fetch as the market falls in unison.

This fear can be seen clearly as Bitcoin’s first sell-off from $19.6k to $6k takes effect, followed by a very slight regaining of hope after the bounce (drop in correlation) before tending back towards 1.0 as we moved for a retest of $6k.

ALT/BTC pairs and hedging

Historically, most Cryptocurrency trading was done with Bitcoin as the base pair, and even now we still see $100Ms daily through Bitcoin pairs. Back in the days where both regulation and market volume was limited, this made a lot of sense. A Crypto-Crypto exchange didn’t need to deal with the hassle of accepting and storing fiat currencies, nor the regulatory issues of handling money.

This had the effect of tying the USD value of altcoins closer to the shifts of Bitcoin, which is still a factor today (although to a lesser extent). This, along with the psychology of fear during a bear market, has lead to the levels of correlation we see in the USD pairs during large pullbacks.

In theory, this makes ALT/BTC pairs extremely useful for trading: during a bull market you’re betting that your chosen coin does better than Bitcoin on its technical merits; during a bear market you expect the ratio to stay relatively level as your coin maintains a correlation near to 1.0 against BTC/USD.

However, in reality we also start to see correlations between the ALT/BTC pairs and Bitcoin also rising during major downturns, suggesting that Bitcoin is used as a safe haven asset to hedge against pullbacks, as people rotate out of higher risk and reward altcoins.

This causes issues in the assumption of a hedged market exposure from trading Bitcoin pairs during a pullback. This does, however, provide yet more signals as to market sentiment. Since you will often see sharp dips in BTC/USD being mirror across ALT/BTC pairs, a deviation from this trend is a helpful indicator of market strength and bullish sentiment.

Trading the correlation

An understanding how different Cryptocurrencies react within the market allows you to optimize portfolios and reduce beta to Bitcoin during downturns, while maintaining upside during the bulk of a bull run.

Well established coins with higher market caps tend to keep high correlations to Bitcoin throughout the market cycle along with lower volatility (at least in terms of crazy Crypto volatility). Meanwhile, low-mid cap coins tend to be used as high risk high reward speculation tools during bull markets, but drop sharply as market sentiment shifts.

The optimal portfolio would likely re-balance periodically between a mix of mid and high cap coins, weighted by a market sentiment metric, while also hedging some exposure by shorting Bitcoin to create synthetic ALT/BTC pairs.

While it is impossible to hedge away all risk in such a new and underdeveloped market, such a portfolio may help to ease the nerves of the more risk averse investor, while maintaining exposure to Cryptocurrency as a whole.

by Matthew Tweed


So You Want to Trade Crypto — Market Cap Distribution and Rise of Altcoins (Part 4)

Bitcoin dominating >90% of the total value of the market to <40%

From the start of 2016 to the end of 2017, we’ve gone from Bitcoin dominating >90% of the total value of the market to <40%.

This flow of capital has lead to a boom in alternative Cryptocurrencies, which offer newer technologies and wider use-cases.

Market Cap Misconceptions

The first point to address is the issue with market capitalization as a metric when applied to Cryptocurrency. For a stock, the market cap is calculated as:

     Price per share * shares outstanding

Which makes sense, as each share represents a stake in the assets and profits of the company. This same calculation is applied to Cryptocurrencies:

     Price per token * tokens available

This starts to cause issues due to the ease that a new token can be made and added to one of the dozens of small exchanges.

If someone creates a new coin with a total supply of 100B and manages to get it listed on a small exchange and trades it a few times with their friends for $1 per coin, it technically has a market cap of $100B. But in reality, is has no true value and no trading volume to sustain any kind of selling pressure.

Maintaining an artificially inflated market cap

Adding to this, there are many coins that do have significant daily trading volumes while maintaining an artificially inflated market cap as the majority the of the supply is locked up by developers and isn’t tradeable. This raises serious questions about how the investors and traders price in total supply of a token and whether the theoretical value of a project lines up with reality.

Many people also misunderstand what market capitalization means in terms of capital flow.

A market cap of $100B does not mean that $100B has been invested into the token, as shown earlier. Nor does a token’s market cap changing from $100B to $150B or $50B mean that $50B of capital has changed hands.

Not enough money in the system to redeem every token

The profit from a Cryptocurrency investment should be treated as “paper gains” until cashed out or hedged — there is simply not enough money in the system to redeem every token to anywhere near the value of its market cap.

Despite this, for a Crypto with sufficient trading volume and age, market cap can be useful for rough comparison, but make sure to always take it with a grain of salt.

Shift in Market Cap Distribution

(Market Cap Values from )

(Market Cap Values from

As we can see, the last few years have not been kind to Bitcoin’s historical dominance of the Crypto market, with many new projects taking off in the first half of 2017.

For many years, Bitcoin has held onto its “first mover advantage”. However, political issues surrounding the development of Bitcoin caused a slow down in advancement — creating a void for a multitude of altcoins to fill.

Result of “ICO Mania”

Over the past year this new crop of development has accelerated, with smart contract platforms taking many of the top spots. 2017 also saw the rise of “ICO Mania”, with dozens of new tokens and projects gaining investment from speculators looking for yet higher yields on their equity.

In the long term, Bitcoin will likely continue its decline in market share, as its older technology simply cannot compete with new offerings. As long as the political issues surrounding development continue, this will not change. Bitcoin made for an excellent proof of concept, but if it can’t adapt, it risks becoming the Myspace of the Crypto world.

Trading the Altcoin Boom

With altcoins making consistent gains in market share and the relative stagnation of Bitcoin development, Bitcoin is likely to drop from the top spot over the next couple of years (if not sooner) in favour of a newer generation Cryptocurrency.

This shift will likely see a huge change in the attitude and composition of the market as a whole, as everyone tries to pile into the new top coin and related technologies, so that they can ride the hype train.

As always, it is best to keep a level head and stick to your trading and investment strategies. A firm understanding of the underlying technology and use-case of a wide range of Cryptocurrencies will serve well in positioning yourself to take advantage of this shift.

There are a wide variety of projects which all have their use-cases

While smart contract focused Blockchains are some of the leaders at the moment, in the long term their value and success will be measured by the applications and businesses that run on top of them. Meanwhile, we shouldn’t forget the other uses of Blockchain, such as ledgers for supply chains, auditing or even Internet of Things devices. There are a wide variety of projects which all have their use-cases.

Cryptocurrency investments should be managed like a stock portfolio. You wouldn’t place your entire value into a single stock, similarly you shouldn’t be overly dear about a single coin. A well balanced holding of different projects across different areas can help hedge against black swan events in the market while profiting from the broad growth of Crypto as an asset class.

by Matthew Twee


So You Want to Trade Crypto - Technical Analysis (Part 3)

A healthy balance between fundamental and technical analysis is necessary when trading any financial product, especially so in Cryptocurrencies, due to both the nature of the asset class and the types of traders in the market.

Technical Analysis of Crypto

Technical analysis in any market is broadly the same, you look for repeating patterns that can be exploited to increase your odds of success. This may be through moving averages, oscillators, support levels or more complex methodologies. All of which serve to reduce the noise in the price data to produce cleaner trading signals.

I’ve seen many arguments about whether or not technical analysis actually produces viable strategies, most of which boil down to the same misunderstanding as to what TA really sets out to achieve.

You will not create a system which can always predict where the market will go. In fact, it will not even be able to predict where the market is likely to go the majority of the time.

Instead, you are looking for system which will deliver signals whose success rate is justified by its risk:reward ratio. The signals may only be correct 10% of the time, but if its profits outsize its losses 10:1, it can still break even.

Technical analysis is not designed to predict the future, it is designed to allow you to understand risk and adapt accordingly. 

Some of the most simple and successful strategies can just take a position aligned with the long term trend and occasionally adjust exposure to keep downside risk at a comfortable level.

Due to the huge bull run Cryptocurrency saw in the past few years, it is important to make sure you back-test and verify strategies during long term downturns too, such as post 2013.

It would be easy to look at the bull run from late 2015 to the start of 2018 and conclude that the most effective method is to “buy the dip” at every opportunity. This only looks at half of the picture, a market always acts in cycles of bullish and bearish periods, whether that be the longer term trend or the short term pullback.

Methods such as the use of an “equity curve” can be used to produce an adaptive trading system, where different strategies will only be trading during favorable market conditions. You may even with to use long term moving average crosses to help differentiate between a pullback and the shift into a new bear/ bull market.

Types of Traders and Investors in Cryptocurrency

Due to the extremely new and highly technical nature of Cryptocurrencies, it has taken a long time for institutions to even acknowledge Crypto, let alone become involved in the market. This has left a market filled with experts in technology rather than the world of finance and trading.

This may be one of the contributing factors to the huge levels of historical volatility, as inexperienced traders and investors playing around with a highly speculative asset are prone to being caught up in the hype and fear cycles of market swings, adding yet more momentum and irrationality to the mix.

In trading, you don’t have to be the smartest in the market (although it can certainly help), simply keeping your head above the emotion of trading cycles and sticking rigidly to a trading strategy is one of the most important factors in success. Emotional trading drives us to FOMO into assets as they near the top before panic selling as it comes crashing back down.

This need for simple and objective decision making is one of the main benefits of algorithmic trading over manual trading. 

A trading bot will not get tired, stressed or emotional. It sticks to the rules of your tried and tested strategy.

Algorithmic trading can be particularly successful in Cryptocurrency due to the level of inefficiency still present. We’re not even past the point where retail traders can still profit from arbitrage strategies — imagine the numbers of other strategies that haven’t been tapped yet!

There are many trading strategies that will give you very limited results in more mature financial markets, such as basic moving average crosses. Every strategy has a limit to the effective capital it can support before it becomes infeasible due to diminishing returns. When everyone knows the same basic strategy, it becomes a race as to who can trade the signals first — and a retail trader can’t compete with a well funded institution.

Final Words

This effect is lessened as the time-period of the strategy increases, but it still pays to find a truly unique strategy, as you will have more opportunity take advantage of its capital limits, rather than fighting to be first on each signal.

by Matthew Tweed


So You Want to Trade Crypto  -  Volatility (Part 2)

The Cryptocurrency market is extremely new and has yet to mature with greater regulation and institutional involvement, leading to sharp moves and high volatility as the market grows. This is a double edged sword — heaven for day traders looking for a bit of excitement while adding significant risk to those wishing to trade longer term.

Historical Volatility

Historical Volatility comparison of asset classes.

  • GVZVIX — CBOE Volatility index for Gold
  • VIX — CBOE Volatility index for the S&P 500
  • EUVIX — CBOE Volatility index for Euro/USD pair
  • BTCVIX — Bitcoin 30 day historical volatility (Bitcoin volatility on right axis as it’s so much higher)

As we can see, there is a clear correlation between the volatility of more traditional traded assets, especially so during events such as the 2009 bounce. Bitcoin, on the other hand, seems to hold limited correlation to any of these other classes of products, even when factoring out the huge difference in average volatility.

Even the volatility of equities, currently at 13.4% from the VIX, is completely dwarfed by that of Bitcoin, coming in at 70% (down from 150% earlier in the year) — and Bitcoin is historically one of the less volatile Cryptocurrencies. Cryptocurrency markets represent a huge shift in gear, incurring more risk while giving the possibility of greater reward.

How to manage the volatility

Managing downside risk is an important part of any successful trading strategy, more so in the extreme volatility of Cryptocurrency. I personally prefer to define a hard stop-loss level prior to each trade, however a “soft” soft-loss/ exit parameter can be sufficient depending on the frequency of the trading strategy.

Very few Crypto exchanges cater to higher frequency trading at the moment, making faster trading strategies more difficult if not impossible in many cases. High frequency market making or scalping strategies are less concerned with hard stop-losses, due to the extremely short exposure to directional risk.

As hold duration for each trade moves into the span of minutes, absolute risk is still relatively limited in most cases. However, as in any market, there are occasional spikes in volatility — but these can be amplified too.

12th Apr 2018 — A move of $1200 — greater than the entire span of previous 2 weeks

12th Apr 2018 — A move of $1200 — greater than the entire span of previous 2 weeks

The risk:reward ratio of each trade is an important factor to a profitable trading system, along with the success rate 

If there is no plan for risk management, a black-swan event (such as the margin cascade seen above) could undo a week’s profits — or even wipe out a while account.

Moving into longer term strategies, such as swing trading

Volatility makes common strategies trickier. The simplest way to reduce the risk on a trade is to reduce position sizing on signals with lower confidence. For example, in forex trading, leverage is widely used to make efficient use of capital on relatively smaller movements. Meanwhile, it’d be madness to take that level of leverage on a Crypto swing trade. Risk doesn’t have to be any higher between asset classes, as long as volatility is adjusted for.

From active trading strategies into indexing, the Cryptocurrency market has provided huge returns over the past couple of years, with the total market capitalization rising from $18.3B at the start of 2017 to $613B by the start of 2018 — over 3000% growth. Unfortunately, past performance isn’t indicative of future returns — it would be impossible to continue at such a rate. Between Jan 2018 and Apr 2018, there was a pullback of 60% from which we’re still recovering.

An automatically re-balanced Cryptocurrency index fund can offer a simple but effective way to gain diversified exposure to the market, but be warned that you will have to be able to stomach the possibility of high drawdowns.

Volatility and Speculative Assets

As I mentioned in the first post of the series, Cryptocurrency cannot be analyzed in the same manner as traditional assets. You have no physical item or stake in a company to which the price is tied. The whole valuation of your token is based on the expectation of future demand, and is thus very susceptible to changes in sentiment.

A stock will be priced above the value of its dividends and assets to account for the expected growth in the business over time. However, during a market downturn, investors become less optimistic about the future as the economy constricts, driving down prices. The business itself may lose some revenue as consumers of the economy have less cash to spend, but there is still a functioning business with value.

Meanwhile, the whole success of Cryptocurrency relies on the ability of a project to attract businesses and developers to use the platform. Speculators are betting that the project will be successful in this goal long term, but there is no tangible asset to retain value if the project falters or market sentiment shifts against it.

This is one of the main reasons why the Cryptocurrency market remains so volatile compared to any other asset class

While this effect will reduce over time, as successful applications and businesses form in the ecosystem, it will not be a quick or easy process.

by Matthew Tweed


50x Faster Bitcoin Price Data Powered by MarketStore for AI Trading

In our last post “How to Setup Bitcoin Historical Price Data for Algo Trading in Five Minutes”, we introduced how to set up bitcoin price data in five minutes and we got a lot of good feedback and contributions to the open source MarketStore.

Photo by  chuttersnap &nbsp;on  Unsplash

Photo by chuttersnap on Unsplash

The data speed is really important

Today, I wanted to tell you how fast MarketStore is using the same data so that you can see the performance benefit of using the awesome open source financial timeseries database.

Faster data means more backtesting and more training in machine learning

Faster data means more backtesting and more training in machine learning for our trading algorithm.  We are seeing a number of successful machine learning-based trading algos in the space, but one of the key points we learned is the data speed is really importantIt's important not just for backtesting, but also for training AI-style algorithms since it by nature requires an iterative process.

This is another post to walk you through step by step. But TL;DR, it is really fast.



Last time, we showed how to setup the historical daily bitcoin price data with MarketStore.

This time, we store all the minute-level historical prices using the same mechanism called background worker, but with a slightly different configuration.


root_directory: /project/data/mktsdb
listen_port: 5993
# timezone: "America/New_York"
log_level: info
queryable: true
stop_grace_period: 0
wal_rotate_interval: 5
enable_add: true
enable_remove: false
enable_last_known: false
 - module:
   on: "*/1Min/OHLCV"
       - 5Min
       - 15Min
       - 1H
       - 1D
 - module:
   name: GdaxFetcher
     query_start: "2016-01-01 00:00"
     base_timefame: “1Min”
       - BTC

Almost 2.5 years with more than 1 million bars

The difference from last time is that background worker is configured to fetch 1-minute bar data instead of 1-day bar data, starting from 2016-01-01.  That is almost 2.5 years with more than 1 million bars. You will need to keep the server up and running for a day or so to fill all the data, since GDAX’s historical price API does not allow you to fetch that many data points quickly.

Again, the data fetch worker carefully controls the data fetch speed in case the API returns “Rate Limit” error. So you just need to sleep on it.

Additional configuration here is something called “on-disk aggregate” trigger.  What it does is to aggregate 1-minute bar data for lower resolutions (here 5 minutes, 15 minutes, 1 hour, and 1 day).

Check the longer time horizon to verify the entry/exit signals

In a typical trading strategy, you will need to check the longer time horizon to verify the entry/exit signals even if you are working on the minute level. So it is a pretty important feature. You would need pretty complicated LEFT JOIN query to achieve the same time-windowed aggregate in SQL. But with MarketStore, all you need is this small section in the configuration file.

The machine we are using for this test is a typical Ubuntu virtual machine with 8 of Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz, 32GB RAM and SSD.

The Benchmark

Unfortunately lots of people in this space are using some sort of SQL database

We are going to have a DataFrame object in python which holds all the minute level historical price data of bitcoin since January of 2016 from the server.  We compare MarketStore and PostgreSQL.

PostgreSQL is not really meant to be the data store for this type of data, but unfortunately lots of people in this space are using some sort of SQL database for this purpose since there is no other alternative.  That’s why we built MarketStore.

The table definition of the bitcoin data in PostgreSQL side looks like this.

btc=# \d prices
              Table "public.prices"
 Column |            Type             | Modifiers
 t      | timestamp without time zone |
 open   | double precision            |
 high   | double precision            |
 low    | double precision            |
 close  | double precision            |
 volume | double precision            |

The code looks like this.

# For postgres
def get_df_from_pg_one(conn, symbol):
    tbl = f'"{symbol}"'
    cur = conn.cursor()
    # order by timestamp, so the client doesn’t have to do it
    cur.execute(f"SELECT t, open, high, low, close, volume FROM {tbl} ORDER BY t")
    times = []
    opens = []
    highs = []
    lows = []
    closes = []
    volumes = []
    for t, open, high, low, close, volume in cur.fetchall():

    return pd.DataFrame(dict(
    ), index=times)

# For MarketStore
def get_df_from_mkts_one(symbol):
    params = pymkts.Params(symbol, '1Min', 'OHLCV')
    return pymkts.Client('http://localhost:6000/rpc'

You don’t need much client code to get the DataFrame object

The input and output is basically the same, in that one symbol name is given, query the remote server over the network, and get one DataFrame.  One strong benefit of MarketStore is you don’t need much client code to get the DataFrame object since the wire protocol is designed to give an array of numbers efficiently.

The Result

First, PostgreSQL

%time df = example.get_df_from_pg_one(conn, 'prices')
CPU times: user 8.11 s, sys: 414 ms, total: 8.53 s
Wall time: 15.3 s

And MarketStore

%time df = example.get_df_from_mkts_one('BTC') 
CPU times: user 109 ms, sys: 69.5 ms, total: 192 ms Wall time: 291 ms 

Both results of course look the same like below.

In [21]: df.head()
                       open    high     low   close   volume
2016-01-01 00:00:00  430.35  430.39  430.35  430.39   0.0727
2016-01-01 00:01:00  430.38  430.40  430.38  430.40   0.9478
2016-01-01 00:02:00  430.40  430.40  430.40  430.40   1.6334
2016-01-01 00:03:00  430.39  430.39  430.36  430.36  12.5663
2016-01-01 00:04:00  430.39  430.39  430.39  430.39   1.9530


50 times difference

A bitcoin was about $430 back then… Anyway, you can see the difference between 0.3 vs 15 seconds which is about 50 times difference. Remember, you may need to get the same data again and again for different kinds of backtesting and optimization as well as ML training.

Also you may want to query not just bitcoins but also other coins, stocks and fiat currencies, since the entire database wouldn’t fit into your main memory usually.

Scalability advantage in MarketStore

MarketStore can serve multiple symbol/timeframe in one query pretty efficiently, whereas with PostgreSQL and other relational databases you will need to query one table at a time, so there is also scalability advantage in MarketStore when you need multiple instruments.

Querying 7.7K symbols for US stocks

To give some sense of this power, here is the result of querying 7.7K symbols for US stocks done as an internal testing.

%time dfs = example.get_dfs_from_pg(symbols) 
CPU times: user 52.9 s, sys: 2.33 s, total: 55.3 s Wall time: 1min 26s 
%time dfs = example.get_dfs_from_mkts(symbols) 
CPU times: user 814 ms, sys: 313 ms, total: 1.13 s Wall time: 6.24 s

Again, the amount of data is the same, and in this case each DataFrame is not as large as the bitcoin case, yet the difference to expand to large number of instruments is significant (more than 10 times).  You can imagine in real life these two (per instrument and multi-instruments) factors multiply the data cost.

Alpaca has been using MarkStore in our production

Alpaca has been using MarkStore in our production for algo trading use cases both in our proprietary customers and our own purposes.  It is actually amazing that this software is available to everyone for free, and we leverage this technology to help our algo trading customers (early access signup is here).

Thanks for reading and enjoy algo trading!



Algo Trading Daily News Scan by Alpaca - May 14th 2018

Algo Trading Daily News Scan by Alpaca - May 14th, 2018

The Top Algo Trading News Stories and Headlines From Around the Web

Algorithmic Trading and TCA Adoption Driven by FX Global Code


“Today the speed of trading, coupled with the number of venues and increasing variety of order types, make it impossible for a human trader to replicate algorithmic behaviour on one order, let alone if the trader has multiple orders to trade simultaneously.”


Algo Traders Warn U.K. Over MiFID II Rules


“Financial firms are pressing the U.K. to stick as closely as possible to the European Union’s MiFID II restrictions on algorithmic trading, warning the Bank of England against imposing even tougher rules on the industry ahead of Brexit.”




“It is a little like a meteorologist looking at two weather systems and seeking to predict how they will change over time, and how they might interact with one another. To stretch the analogy, when you try to decide on the best currency pair for Forex trading, you are selecting two out of more than 100 different weather systems to pitch against one another.”


Why Algorithmic Trading Really Works


“People often ask how I got my start. They typically want to know how I became a ‘system trader’ — it’s not the sort of job you hear about at a school careers night.”


How to become an algo trader


“Knowing the data structures, algorithms and C++ or another programming language inside and out is critical. That attribute is not easily available, and people really value that. Most [financial services and fintech] firms will hire a great programmer without trading experience over a mediocre programmer who knows trading.”


Automated Trading Broadens Accessibility to Crypto


“While traditional asset markets like stocks have set trading hours and no weekend trading, cryptocurrency prices move around the clock, every day of the year. For most investors, staying alert and monitoring an investment portfolio around the clock is not just unfeasible, but also impractical. While automated trading has been around for years, it was largely inaccessible for most ordinary traders, remaining in the purview of sophisticated investors until this point. However, in keeping with the democratization of finance instilled by cryptocurrency, algorithmic trading is being simplified for mass-market consumption.”


How to Setup Bitcoin Historical Price Data for Algo Trading in Five Minutes

Alpaca platform is now accepting signups for our algo trading waitlist and we do believe trading automation is the future. 

Photo by  Chris Liverani  on  Unsplash

One of the great benefits of algorithmic trading is that you can test your trading strategy against historical data.

Especially for new strategies you developed on your own, you don’t really know how it will perform without testing it against reliable data. Algo trading is not that different from software development. Today, most good software code is continuously tested. As the code evolves, developers continually test it against real use cases to make sure that alterations won’t result in future failures. You want to test your algo when you drop the very first version. But you also want to test the algo when you make changes or adjustments while the algo is running.

Here is one problem you face; data. 

While it is always good to test as many angles as possible, running a number of backtests is going to be a very data-intensive workload that requires access to enough data to have visibility into a very long history. This is why you cannot run this iteration using GDAX API directly. You need to store the data somewhere for your own purposes.

The Tutorial

So the first question to come to mind is always how to get the data and prepare it for successful backtesting. Well, today I am going to tell you how to use MarketStore to acquire a long history of Bitcoin price data for this purpose of running the most accurate backtest possible. And this setup tutorial is going to be quick. Don’t waste your time setting up this and that. All you need is:

  • docker (either on Windows, Mac or Linux),

  • a console terminal,

  • and a big cup of strong coffee! ☕

Got’em? Alright, let’s get started.



Medium - bitcoin data1.png

Here is the high level picture of today’s system. We will start a MarketStore instance using docker container, and run a background worker that calls GDAX price API so that we can pull the bitcoin historical price from their endpoint quickly and make it available for backtest clients to query over HTTP.

We will start another container for the client using python anaconda with python3 image. We use the official client package named pymarkestore. You will get a DataFrame from MarketStore.

Setup MarkeStore Server

There is the official build of MarketStore docker image today publicly available in DockerHub, but first, let’s write a config file for the server.

In the github repository you can find an example config file in YAML format: but I’m putting our example here.

root_directory: /project/data/mktsdb   listen_port: 5993   log_level: info   queryable: true   stop_grace_period: 0   wal_rotate_interval: 5   enable_add: true   enable_remove: false   enable_last_known: false   bgworkers:     - module:       name: GdaxFetcher       config:         symbols:          - BTC          base_timeframe: "1D"          query_start: "2018-01-01 00:00"

This configures the server so that it fetches the GDAX historical price API for 1-day bars since 2018–01–01. Save this config as $PWD/mkts.yml file. The server listens on the port 5993 as default. Now let’s bring up the server.

$ docker run -v $PWD/mktsdb:/project/data/mktsdb -v $PWD/mkts.yml:/tmp/mkts.yml --net host alpacamarkets/marketstore:v2.1.1 marketstore -config /tmp/mkts.yml

The server should automatically download the docker images from DockerHub if you haven’t, and start the server process with the config. Hopefully, you will see something like this.

I0430 05:54:56.091770       1 log.go:14] Disabling "enable_last_known" feature until it is fixed... I0430 05:54:56.092200       1 log.go:14] Initializing MarketStore... I0430 05:54:56.092236       1 log.go:14] WAL Setup: initCatalog true, initWALCache true, backgroundSync true, WALBypass false: I0430 05:54:56.092340       1 log.go:14] Root Directory: /project/data/mktsdb I0430 05:54:56.097066       1 log.go:14] My WALFILE: WALFile.1525067696092950500.walfile I0430 05:54:56.097104       1 log.go:14] Found a WALFILE: WALFile.1525067686432055600.walfile, entering replay... I0430 05:54:56.100352       1 log.go:14] Beginning WAL Replay I0430 05:54:56.100725       1 log.go:14] Partial Read I0430 05:54:56.100746       1 log.go:14] Entering replay of TGData I0430 05:54:56.100762       1 log.go:14] Replay of WAL file /project/data/mktsdb/WALFile.1525067686432055600.walfile finished I0430 05:54:56.101506       1 log.go:14] Finished replay of TGData I0430 05:54:56.109380       1 plugins.go:14] InitializeTriggers I0430 05:54:56.110664       1 plugins.go:42] InitializeBgWorkers I0430 05:54:56.110742       1 log.go:14] Launching rpc data server... I0430 05:54:56.110800       1 log.go:14] Launching heartbeat service... I0430 05:54:56.110822       1 log.go:14] Enabling Query Access... I0430 05:54:56.110844       1 log.go:14] Launching tcp listener for all services...

If you see something like “Response error: Rate limit exceeded”, that’s a good sign, not a bad one, since it means the background worker successfully fetched the price data and reached to rate limit. The fetch worker will suspend for a while and restart to catch up to the current price automatically. You just need to keep it running.

Client Side

MarketStore implements JSON-RPC and MessagePack-RPC for query. MessagePack-RPC is particularly important for performance of a query on a large dataset. Thankfully, there is already python and go client library so you don’t have to implement the protocol. In this article, we use python. We start from miniconda3 image from another terminal.

$ docker run -it --rm -v $PWD/ --net host continuumio/miniconda3 bash # pip install ipython pymarketstore

We have installed ipython and pymarketstore, including their dependencies. From this terminal, let’s start an ipython shell and query MarketStore data.

# ipython (base) root@hq-dev-01:/# ipython Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) Type 'copyright', 'credits' or 'license' for more information IPython 6.3.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import pymarketstore as pymkts
In [2]: param = pymkts.Params('BTC', '1D', 'OHLCV', limit=100)
In [3]: df = pymkts.Client('http://localhost:5993/rpc').query(param).first().df()
In [4]: df[-10:] Out[4]:                               Open     High      Low    Close Volume Epoch 2018-04-14 00:00:00+00:00  7893.19  8150.00  7830.00  8003.11   9209.196953 2018-04-15 00:00:00+00:00  8003.12  8392.56  8003.11  8355.25   9739.103514 2018-04-16 00:00:00+00:00  8355.24  8398.98  7905.99  8048.93  13137.432715 2018-04-17 00:00:00+00:00  8048.92  8162.50  7822.00  7892.10  10537.460361 2018-04-18 00:00:00+00:00  7892.11  8243.99  7879.80  8152.05  10673.642535 2018-04-19 00:00:00+00:00  8152.05  8300.00  8101.47  8274.00  11788.032811 2018-04-20 00:00:00+00:00  8274.00  8932.57  8216.21  8866.27  16076.648797 2018-04-21 00:00:00+00:00  8866.27  9038.87  8610.70  8915.42  11944.464063 2018-04-22 00:00:00+00:00  8915.42  9015.00  8754.01  8795.01   7684.827002 2018-04-23 00:00:00+00:00  8795.00  8991.00  8775.10  8940.00   3685.109169

Voila! You just got the daily bitcoin price in hand in the DataFrame format. Note the second line (param = …) determines which symbol and timeframe to query, with some query predicates such as the number of rows or date range to query. From here, you can do a number of things including calculating indicators such as moving average and bollinger band, or find the statistical volume anomaly using some scipy package.


I want to emphasize that it is very important to build a performant historical dataset to study and develop a trading algorithm, and you can do it quickly with MarketStore as we have just walked through. This article demonstrated how to work with the bitcoin prices from GDAX, but you can hook up other data sources as well pretty easily using pymarketstore’s write method. You can also write your own custom background data fetcher.

Again, the query performance is going to be critical when in comes to backtesting, since you want to iterate quickly to get the results. now You may wonder how fast MarketStore can be. I will show the lightning fast query speed with huge data set in the next post.

In the meantime, please leave any questions in the comments or ask @AlpacaHQ regarding this tutorial. Leave your email below so we can notify you when we can grant access to the full trading platform! You can also check us out at

Happy algo trading!