Blueprint For A Momentum Strategy

  1. Models
  2. Framework
    • Statics
    • Historical Data (OHLCV)
    • Live Prices
    • Indicators
    • Charts
  3. An Implementation In Java
  4. Data Analytic Tools
  5. Market Access Tools
    • Chinese Futures Markets: CTP Java Wrapper Via Swig
    • Chinese Futures Markets: PythonGo
    • US Futures Markets (CTA): IBKR TWS API in Java
    • ArrayManager: A Model From VNPY and PythonGo
  6. Tools For Backtests
  7. Reports

20240307: Forming Bottom

After Lunar New Year, the market has crashed and has been recovering. On 6 Mar 2024 (Wed), all indicators suggested by the strategy suggest to move onto risk assets.

The model allocated the first stock 601600 (大秦铁路). Interesting, this is the value stock that I’ve been tracking, who pays consistent dividend over the past years. Nowadays, one norm of the market is on the value investing, where the fundamental logic might not be the risk on, but instead, the rate cut that’s happening right now.

The fact that the bond market is running bull is due to the rate cut. With the discounting rate decreasing, the bond is benefiting directly and price is constantly moving up. The 5y and 10y treasury bonds that I’ve tracked in CTA model have moved up considerable.

The value stocks who pay dividends consistently benefited from rate cut as well. Such stocks include SOE banks, coal mining groups, railway companies, and etc. From last Nov till this Mar, both banking and mining stocks have increased considerable in market prices.

US market is fully allocated; the model is lagging behind the index, but it’s trending up. I would expect that US stock market would have turbulence when the election starts, but I will let the model to work it out, and I just need to track it closely.

Logs For Year 2024

The start of year Dragon is dramatic in stock market in China, where the index crashed to 2635.09 and pulled back to above 2800 before the nation entered into Chinese New Year holiday season.

The drama has erased lots of wealth for those who invested in snowball products linked to ZZ1000 stocks; and lots of quant fund had huge revaluation on their AUM. The chairman of CSRC was replaced by a new guy, who is supposed to save investors from the crisis.

After Chinese New Year, the stock market appears to be climbed out of the crisis and edged above 3000, while the rest of the world continue to enjoy the bull run on risk assets, including US, Europe, Japan, Vietnam and etc.

US would pause rates hikes and might cut rates if inflation is under control; emerging markets could receive investments as dollar rates tops and dollar index starting to decline.

I would expect that both US and Chinese market could offer good opportunities for stock markets. As the start of this year, my portfolio is fully allocated for US stocks.

Python Parses Excel Table To DataFrame

Scenario

A broker statement is an excel file containing different section of data in tabular format. I’m interested to load the data to dataframe for analysis and execute a query into risk database for other interesting analysis tasks.

Technical breakdown

  • xlwings – a python library to access excel sessions.
  • sqlalchemy – a python library to access oracle.

xlwings

The steps of processing a statement, includes: open an excel, locate the table in interest, load the table to dataframe, process the dataframe, and close the excel.

#given an entire column, trying to get the cell that contains the text.
#e.g. cell=find(sheet.range("B:B"),"(reports of P/L)")
def find_cell(myrange, what):
    retval = None
    for cell in myrange:
        if cell.value==what:
            retval = cell
            break
    return retval

#locate the sheet for futures.
#1. QFII: the table is in sheet RQFII
#2. normal: if there's no QFII details, then it should be a domestic account, usually the first page.
def find_sheet(book):
    retval = None
    for sheet in book.sheets:
        if "RQFII" in sheet.name and "Future" in sheet.name:
            retval = sheet
            break
    if retval is None:
        retval=book.sheets[0]
    return retval

############################################################
#CLOSE OUT DETAILS
#load closeout details section from the broker statement has different format.
#each broker has it's own reader to extract the table into dataframe.
#steps to load
#a. load the workbook 
#b. search the table heading to locate the row, with row_num
#c. select the row_num+1 and expand selection
#d. return dataframe from selected range. 
def closedetails(indir,infile,broker):
    stmtfile="{}{}".format(indir,infile)
    with xw.App() as app:
        #load workbook 
        print("loading... {}".format(stmtfile))
        book = xw.books.open(stmtfile)
        #identify the correct sheet, QFII 
        sheet=find_sheet(book)
        #find the row.
        name_range=m_tbl_column[broker]
        name_table=m_tbl_name[broker]
        table_cell = find_cell(sheet.range(name_range), name_table)
        row_num = table_cell.row
        print(f"file={stmtfile}, broker={broker}, range={name_range}, title={name_table}, row={row_num}")
        #loadthe range into dataframe
        rng_closeout=sheet['A'+str(row_num+1)].expand()
        df_closeout=rng_closeout.options(pd.DataFrame).value
        print("closeout total rows: {}".format(len(df_closeout)))
        #check the open trade date; and we only want to re-upload trades before 20Nov.
        name_opendate=m_tbl_opendate[broker]
        df_closeout['OpenDate']=pd.to_datetime(df_closeout[name_opendate],format='%Y%m%d')
        #now export the file to excel
        df_closeout_pre=df_closeout[df_closeout.OpenDate<'2023-11-20'] #under old structure
        df_closeout_pos=df_closeout[df_closeout.OpenDate>='2023-11-20']#under package structure
        print("closeout involves trade before 20Nov: count={},opencount={}".format(len(df_closeout_pre),len(set(df_closeout_pre[m_tbl_opennumber[broker]]))))
        print("closeout invovles trade after  20Nov: count={},opencount={}".format(len(df_closeout_pos),len(set(df_closeout_pos[m_tbl_opennumber[broker]]))))
        if len(df_closeout_pre)>0:
            print(f"indir:{indir}")
            print(f"infile:{infile}")
            df_closeout_pre.to_excel('{}_RELOAD_{}'.format(indir,infile))
            return df_closeout_pre

sqlalchemy

sql_exec=sql_open_template.format(broker,stdate,account,account,",".join(["'ECN{}'".format(x) for x in list(set(opens))]))

export_fn_sql=f'{broker}_{stdate}_{entity}.sql'
print("generating sql file {} for analysis...".format(export_fn_sql))
with open(export_fn_sql,'w') as outfile:
    outfile.write(sql_exec)


print("executing oracle query...")
#read oracle data into dataframe 
# with cx_Oracle.connect(username,username,dsn=conn_string) as conn:
# df_open_trades=pd.read_sql(sql=sql_opens,con=conn)
engine=sqlalchemy.create_engine(f"oracle+cx_oracle://{username}:{username}@{hostname}:{port}/?service_name={servicename}",arraysize=1000)
df_open_trades=pd.read_sql(sql_exec,engine)
print("from statement, count={}, loaded={}\n".format(len(opens),len(df_open_trades)))

if exportfile:
    fn_export_tradecount=f'tradecount_{context_dt}.xlsx'
    print("exporting file: {} ...".format(fn_export_tradecount))
    df_open_trades.to_excel(fn_export_tradecount,index=False)

print("oracle returns count={}".format(len(df_open_trades)))

Properly Start Mina Ftp Server From Issues

Default package from apache is broker, that you will have error when launching ftp server following its guide when running on windows.

Error Message

org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 26 in XML document from file [C:\scratch\softwares\apache-ftpserver-1.2.0-bin\apache-ftpserver-1.2.0\res\conf\ftpd-typical-traf.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 26; columnNumber: 4; cvc-elt.1.a: Cannot find the declaration of element 'server'.
        at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:404)
...
Caused by: org.xml.sax.SAXParseException; lineNumber: 26; columnNumber: 4; cvc-elt.1.a: Cannot find the declaration of element 'server'.

This config file gives me going.

<?xml version="1.0" encoding="UTF-8"?>
<server xmlns="http://mina.apache.org/ftpserver/spring/v1"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
	   http://mina.apache.org/ftpserver/spring/v1 http://mina.apache.org/ftpserver/ftpserver-1.0.xsd	
	   "
	id="myServer">
	<listeners>
		<nio-listener name="default" port="2121">
		    <ssl>
                <keystore file="./res/ftpserver.jks" password="password" />
            </ssl>
		</nio-listener>
	</listeners>
	<file-user-manager file="./res/conf/users.properties" />
</server>

Windows cmd does not allow ftp to connect to customized port, while mina runs on 2121. But you can do the following to work around it.

H:\>ftp
ftp> open localhost 2121
User:
Password:
ftp>binary
ftp>get somefile.zip
ftp>bye

20231127: Expecting Rates Stale Or Cut

The FED dot plot as of Sep 2023 (appendix 1) shows that rates hike is ending, and expectation is that there’ll be cut in second half of 2024. As recently dollar index starts dropping, world wide stock markets are bouncing up higher.

Over the past a few months, there has been such expectations (rate->dollar strength->commodities) following which the expectations were invalidated. CTA strategies were having drawdowns. Maybe at the turning point, CTA would have such issue when expectation is mixed and not stable.

The Chinese stock market remains weak, after the authority publishes measures and policies to encourage investing. Investors, foreign or domestic, institutional or retail, are coming to realize that the Chinese stock market is a tool for enterprise to cash out rather than giving investors incentives after taking risks. The recent policies try to stop enterprise from cashing too much, which helps to ease the bleeding market. It appears that all the measures that the government delivered have helped market from crashing further, but gives no strength to lift the market, since the macroeconomy is still weak. Appendix 4 shows the divergence of HS300 from SP500, which shows that the recent Xi’s visit to US shows no pushing factor to the market.

The US stock market is picking strength (Appendix 3), after it’s recent adjustment. Given the optimistic rate expectation and seasonal sentiment, the market continues edge higher, which leads to the momentum strategy to start allocating risks to stocks (Appendix 2).

Appendix 1: Dot Plot From FED as of 20 Sep 2023

Appendix 2: Risk Allocation From Momentum Strategy as of 27 Nov 2023

Appendix 3: SP500 Index Shows Strength

Appendix 4: SP500 vs HS300

CLI Python Argparse

Python provides a way to build command which works like bash but you have leverage to python.

python pvt.py --help
python pvt.py tradedump out

import pandas as pd
import argparse

parser  = argparse.ArgumentParser()
parser.add_argument("tradedump", help="provide csv file of the external trade dump.")
parser.add_argument("out", help="output file names for pivot results.")
args    = parser.parse_args()
#defines input parameters
#path to trade dump
tradedump   = args.tradedump
output      = args.out

export_path=r"path/to/directory"
out_instrument  = export_path + f"\{output}_pvt_instruments.xlsx"
out_counterprt  = export_path + f"\{output}_pvt_counterparts.xlsx"
out_entities    = export_path + f"\{output}_pvt_entities.xlsx"

Reference:

MDES: Upload OHLC CSV Via Curl

From start.spring.io, generate project for web, thymeleaf, and jpa, and add h2 dependencies to it.

Data Model

public class Student {
	private long id;
	private String name;
	private String email;
	public Student() {}
	
	public Student(long id, String name, String email) {
		this.id = id;
		this.name = name;
		this.email = email;
	}
...

DAO Services

@Mapper
public interface StudentsMapper {
	@Select("SELECT * FROM students where id=#{id}")
	public Student find(int id);
	
	@Select("SELECT * FROM students")
	public List<Student> findAll();
	
	@Update("MERGE INTO students KEY(id) VALUES (#{id},#{name},#{email})")
	public Integer insertOrUpdate(Student student);
	
	@Delete("DELETE FROM students WHERE id=#{id}")
	public void delete(int id);
}

public interface StudentService {
	public Student find(int id);
	public Student find(String id);
	public Student saveUpdateStudent(Student student);
	public Student delete(int id);
	public Student delete(String id);
}

Controllers

	@RequestMapping(value="/csv", consumes="text/plain", produces=MediaType.APPLICATION_JSON_VALUE)
	public ResponseEntity<String> uploadCsv(@RequestBody String input){
		logger.info("input={}", input);
		return ResponseEntity.ok().build();
	}

curl -X POST --data-binary @student.txt -H "Content-Type: text/plain" http://localhost:8080/students/csv

	@PostMapping("/fileupload")
	public ResponseEntity<List<Student>> fileUpload(@RequestPart("file") MultipartFile file){
		List<Student> retval = Lists.newArrayList();
		
		List<String> lines;
		try {
			lines = ByteSource.wrap(file.getBytes()).asCharSource(Charsets.UTF_8).readLines();
			FluentIterable.from(lines).forEach(x->logger.info(x));
		} catch (IOException e1) {
			e1.printStackTrace();
		}
		try {
			 lines = ByteSource.wrap(file.getBytes()).asCharSource(Charsets.UTF_8).readLines();
			for(String line : lines) {
				retval.add(new Student.Builder().from(line).build());
			}
		} catch (IOException e) {
			e.printStackTrace();
		}
		//update students
		for(Student item : retval) {
			service.saveUpdateStudent(item);
		}
		if(retval.size()==0) {
			return ResponseEntity.noContent().build();
		}else {
			return ResponseEntity.ok(retval);
		}
	}	

curl -v http://localhost:8080/students/fileupload -X POST -F "file=@student.txt" -H "Content-Type: multipart/form-data"

Query Account With Time Lags

To track positions timely, I’m checking positions whenever there’s a trade done via onTrade() method from PythonGo. However there’s a threshold on how frequent you could query the account, and you might receive warning from the broker if you query too often.

The solution to this issue is to place a time lag from current query to the next, and we would use the cached positions. At time T, before we query account, we would check Redis if the Position-Key has value, if so, we will use the cached positions; if not, then we have to query account to have a list of positions, which would be saved to Redis under Position-Key, and such key would expire in N seconds.

Related redis commands as following.

SET mkey "hello"
GET mkey
EXPIRE mkey 10
TTL mkey
GET mkey

import redis
r=redis.Redis(host='localhost',port=6379,db=0)
r.set("mkey","hello")
r.get("mkey")
r.expire("mkey")
r.ttl("mkey")
r.get("mkey")

Messaging between PythonGo and external apps

PythonGo has packaged lots of functions within the software itself. It is hard to directly code and debug the application, but it does offer the access to Python, so that you can code anything from collecting tick data to complex strategies. This entry tries to show how technically we can publish and subscribe messages via python, which is helpful for us to enable PythonGo for such messaging later on.

Listener class

class MessageListener:
    def __init__(self,name,channel):
        self.name=name
        self.channel=channel
    def listen(self,r):
        if r:
            listener = r.pubsub()
            listener.subscribe(self.channel)
            for message in listener.listen():
                print(f"{self.name} has message: {message}")
        else:
            print('please create redis connection.')
    #build a function to unsubscribe as well to terminate threads.

Listener Thread

import threading
import redis
r=redis.Redis(host='localhost', port=6379, db=0)
listener1=MessageListener('l1','channel1')
listener2=MessageListener('l2','channel1')
t1=threading.Thread(target=listener1.listen,args=(r,))
t2=threading.Thread(target=listener2.listen,args=(r,))
t1.start()
t2.start()

Publisher

import redis
r=redis.Redis(host='localhost', port=6379, db=0)
r.publish('channel1',"hello from publisher")

Reference:

CURL in Windows

To post json in windows, CURL needs to have special treatment because single quote does not work.

1 JSON via command line 1.1 JSON quoted with escape character

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{\"name\": \"value\"}" http://localhost:8080/process

1.2 JSON quoted with another double quote

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{""name"": ""value""}" http://localhost:8080/process

2 JSON via text file

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @json.txt http://localhost:8080/process

On the same directory, create a file “json.txt” with following content.

{"name": "value"}

Reference: https://stackoverflow.com/questions/11834238/curl-post-command-line-on-windows-restful-service