Tag: JSON

CURL in Windows

To post json in windows, CURL needs to have special treatment because single quote does not work.

1 JSON via command line 1.1 JSON quoted with escape character

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{\"name\": \"value\"}" http://localhost:8080/process

1.2 JSON quoted with another double quote

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{""name"": ""value""}" http://localhost:8080/process

2 JSON via text file

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @json.txt http://localhost:8080/process

On the same directory, create a file “json.txt” with following content.

{"name": "value"}

Reference: https://stackoverflow.com/questions/11834238/curl-post-command-line-on-windows-restful-service

Source 60m Bar Via Java

I’ve decided to source 60m bars for futures contracts via Sina Finance for building golden source of market data, as described here.

The source provides data in Json, which can be easily converted into Objects via jacksonxml library, with a few tricks.

Trick 1. regexp
The response contains overheads mixed with JSON, so we need to strip it off in order to obtain the clean and tidy JSON. The following code sniplet takes Json content for OHLCVP.

	@Test
	public void test_read(){
		CharSource source = Files.asCharSource(new File(FILESTORAGE + File.separator + HISTORY), Charsets.US_ASCII);
		try {
			//encoding conversion is just taking too long to try. let's focus on SecurityHistory conversion
			List<String> lines = source.readLines();
			logger.error("count: {}", lines.size());
			for(String line : lines) {
				logger.error("line={}", line.length());
			}

			Pattern p = Pattern.compile("(.*)(\\Q=(\\E)(.*)\\Q);\\E");
			Matcher m = p.matcher(lines.get(1));
			if(m.find()){
				int count = m.groupCount();
				logger.error("count:{}", count);
				for(int i=0;i<=count;i++) {
					logger.error("token:{}", m.group(i));
				}
			}

2. jackson
ObjectMapper can parse the JSON string to objects conveniently, via the following function call:
mapper.readValue(json, new TypeReference<List<SinaFutureXminBarJson>>(){});
However, we need to define SinaFutureXminBarJson model class to hold data from json text, which should define a deserializer, JsonDateTimeDeserializer, which helps to convert string, such as “2020-10-15 22:00:00” to Joda object of LocalDateTime. 

			String json = m.group(3);
			ObjectMapper mapper = new ObjectMapper();
			mapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
			mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
			mapper.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));//2017-07-10 00:00:00

			try {
				List<SinaFutureXminBarJson> items = mapper.readValue(json, new TypeReference<List<SinaFutureXminBarJson>>(){});
				List<SinaFutureXminBar> retval = Lists.newArrayList();
				for(SinaFutureXminBarJson bar : items) {
					retval.add(new SinaFutureXminBar.Builder().build(bar));	
				}
				for(SinaFutureXminBar bar : retval) {
					logger.error("bar: {}", bar);
				}
			} catch (JsonParseException e) {
public class SinaFutureXminBarJson {
	@JsonDeserialize(using = JsonDateTimeDeserializer.class)
	private LocalDateTime d;
	private Double o;
	private Double h;
	private Double l;
	private Double c;
	private Long v;
	private Long p;

	public LocalDateTime getD() {
		return d;
	}
	public void setD(LocalDateTime d) {
		this.d = d;
	}
	public Double getO() {
		return o;
	}
	public void setO(Double o) {
		this.o = o;
	}
	public Double getH() {
		return h;
	}
	public void setH(Double h) {
		this.h = h;
	}
	public Double getL() {
		return l;
	}
	public void setL(Double l) {
		this.l = l;
	}
	public Double getC() {
		return c;
	}
	public void setC(Double c) {
		this.c = c;
	}
	
	public Long getV() {
		return v;
	}
	public void setV(Long v) {
		this.v = v;
	}
	public Long getP() {
		return p;
	}
	public void setP(Long p) {
		this.p = p;
	}
	public String toString() {
		return MoreObjects.toStringHelper(this.getClass())
			.add("d", d)
			.add("o", o)
			.add("h", h)
			.add("l", l)
			.add("c", c)
			.add("v", v)
			.add("p", p)
			.toString();
	}
}
public class JsonDateTimeDeserializer extends StdDeserializer<LocalDateTime>{
    private static final long serialVersionUID = 1L;
    private static DateTimeFormatter format = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss");

    public JsonDateTimeDeserializer() {
        this(null);
    }

    public JsonDateTimeDeserializer(Class<LocalDateTime> t) {
        super(t);
    }

    @Override    
    public LocalDateTime deserialize(JsonParser parser, DeserializationContext context)
            throws IOException, JsonProcessingException {

        String date = parser.getText();

        return format.parseDateTime(date).toLocalDateTime();
    }
}

BeanShell Limitations On Generics

BeanShell does not support Generics, so I have to research alternative ways of parser utility.

To parse a list of objects from a JSON string, there’re two approaches, either to read it into a List of objects (which requires Generics) or into an array of objects. BeanShell would require the latter approach.

Firstly, we could use TypeReference to read a list of objects directly, with conversion function code snaplet as below. This code works perfectly in Java, however, it breaks in BeanShell, since Generics is not supported in BeanShell. 

public List<SecurityHistory> apply(String input) {
    List<SecurityHistory> retval = Lists.newArrayList();

    ByteSource source = CharSource.wrap(input).asByteSource(Charsets.UTF_8);

    ObjectMapper m = new ObjectMapper();
    m.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
    m.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
    m.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));//2017-07-10 00:00:00
    try {
        List<CsiHistory> items = m.readValue(source.openStream(), new TypeReference<List<CsiHistory>>(){});
        for(CsiHistory item : items){
            retval.add(new SecurityHistory.Builder()
                .code(item.getIndx_code())
                .stype(EInstrumentType.INDEX)
                .open(item.getTclose())
                .high(item.getTclose())
                .low(item.getTclose())
                .close(item.getTclose())
                .historyDate(item.getTradedate())
                .build());
        }
    } catch (JsonParseException e) {
        e.printStackTrace();
    } catch (JsonMappingException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }

    return retval;
}

Secondly, Jackson allows to use array to get Jackson to parse the string. Personally, I prefer to use List structure for the parsing, however there’s no choice if I need BeanShell to run the code. This approach works in BeanShell since it doesn’t use any Generics.

public SecurityHistory[] apply(String input) {
	SecurityHistory[] retval = new SecurityHistory[]{};
	ByteSource source = CharSource.wrap(input).asByteSource(Charsets.UTF_8);
	ObjectMapper m = new ObjectMapper();
	m.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
	m.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
	m.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));//2017-07-10 00:00:00
	try {
		CsiHistory[] items = m.readValue(source.openStream(), CsiHistory[].class);
		//initialize return value
		retval = new SecurityHistory[items.length];
		
		for(int i=0; i<items.length; i++){
			CsiHistory item = items[i];
			retval[i] = new SecurityHistory.Builder()
				.code(item.getIndx_code())
				.stype(EInstrumentType.INDEX)
				.open(item.getTclose())
				.high(item.getTclose())
				.low(item.getTclose())
				.close(item.getTclose())
				.historyDate(item.getTradedate())
				.build();
		}
	} catch (JsonParseException e) {
		e.printStackTrace();
	} catch (JsonMappingException e) {
		e.printStackTrace();
	} catch (IOException e) {
		e.printStackTrace();
	}
	return retval;
}

Reference:
1. Convert Array Objects: https://makeinjava.com/convert-array-objects-json-jackson-objectmapper/

FasterXML Jackson Parses byte[] But Fails String

As described in index component analysis, sina finance gives a list of live ticks in format of JSON. One way to work with JSON is to tokenize and extract prices in interest and ignores the rest; the other way is to use Serializer to convert JSON into java object model for easy access.

FasterXML Jackson library is a good choice to try, as I did. However, there’s some catches to get the ball rolling. As I did the code in TDD approach, a test case was created to test if the serialization works. It turns out that parsing String fails with exceptions, while parsing byte[] of the String yields pretty results.

1. The JSON text is like following:

[{symbol:"sh600111",code:"600111",name:"北方稀土",trade:"13.390",pricechange:"0.000",changepercent:"0.000",buy:"13.390",sell:"13.400",settlement:"13.390",open:"13.410",high:"13.420",low:"13.260",volume:30703541,amount:409573568,ticktime:"15:00:00",per:535.6,pb:5.523,mktcap:4864675.374,nmc:4864675.374,turnoverratio:0.84511},
{symbol:"sh600139",code:"600139",name:"西部资源",trade:"6.730",pricechange:"0.020",changepercent:"0.298",buy:"6.720",sell:"6.730",settlement:"6.710",open:"6.710",high:"6.760",low:"6.680",volume:3920791,amount:26339581,ticktime:"15:00:00",per:384.571,pb:5.341,mktcap:445452.311884,nmc:445452.311884,turnoverratio:0.59236}]

2. Object model is created based on JSON parameters

class IndexComponentItem {
    private String symbol;//:"sh600000",
    private String code;//:"600000",
    private String name;//:"浦发银行", ,<-company name in Chinese
    private double trade;//:"16.210",
    private double pricechange;//:"0.140",
    private double changepercent;//:"0.871",
    private double buy;//:"16.220",
    private double sell;//:"16.230",
    private double settlement;//:"16.070",
    private double open;//:"16.070",
    private double high;//:"16.230",
    private double low;//:"16.040",
    private double volume;//:12262167,
    private double amount;//:197653917,
    private Date ticktime;//:"15:00:00",
    private double per;//:6.083,
    private double pb;//:1.062,
    private double mktcap;//:35043231.753562,
    private double nmc;//:33261004.880897,
    private double turnoverratio;//:0.05976
	public String getSymbol() {
		return symbol;
	}
	public void setSymbol(String symbol) {
		this.symbol = symbol;
	}
	public String getCode() {
		return code;
	}
	public void setCode(String code) {
		this.code = code;
	}
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}
	public double getTrade() {
		return trade;
	}
	public void setTrade(double trade) {
		this.trade = trade;
	}
	public double getPricechange() {
		return pricechange;
	}
	public void setPricechange(double pricechange) {
		this.pricechange = pricechange;
	}
	public double getChangepercent() {
		return changepercent;
	}
	public void setChangepercent(double changepercent) {
		this.changepercent = changepercent;
	}
	public double getBuy() {
		return buy;
	}
	public void setBuy(double buy) {
		this.buy = buy;
	}
	public double getSell() {
		return sell;
	}
	public void setSell(double sell) {
		this.sell = sell;
	}
	public double getSettlement() {
		return settlement;
	}
	public void setSettlement(double settlement) {
		this.settlement = settlement;
	}
	public double getOpen() {
		return open;
	}
	public void setOpen(double open) {
		this.open = open;
	}
	public double getHigh() {
		return high;
	}
	public void setHigh(double high) {
		this.high = high;
	}
	public double getLow() {
		return low;
	}
	public void setLow(double low) {
		this.low = low;
	}
	public double getVolume() {
		return volume;
	}
	public void setVolume(double volume) {
		this.volume = volume;
	}
	public double getAmount() {
		return amount;
	}
	public void setAmount(double amount) {
		this.amount = amount;
	}
	public Date getTicktime() {
		return ticktime;
	}
	public void setTicktime(Date ticktime) {
		this.ticktime = ticktime;
	}
	public double getPer() {
		return per;
	}
	public void setPer(double per) {
		this.per = per;
	}
	public double getPb() {
		return pb;
	}
	public void setPb(double pb) {
		this.pb = pb;
	}
	public double getMktcap() {
		return mktcap;
	}
	public void setMktcap(double mktcap) {
		this.mktcap = mktcap;
	}
	public double getNmc() {
		return nmc;
	}
	public void setNmc(double nmc) {
		this.nmc = nmc;
	}
	public double getTurnoverratio() {
		return turnoverratio;
	}
	public void setTurnoverratio(double turnoverratio) {
		this.turnoverratio = turnoverratio;
	}
	public String toString(){
		return String.format("{symbol:\"%s\",code:\"%s\",name:\"%s\",trade:\"%f\",pricechange:\"%f\",changepercent:\"%f\",buy:\"%f\",sell:\"%f\",settlement:\"%f\",open:\"%f\",high:\"%f\",low:\"%f\",volume:%f,amount:%f,ticktime:\"%tT\",per:%f,pb:%f,mktcap:%f,nmc:%f,turnoverratio:%f}",
			    symbol,
			    code,
			    name,
			    trade,
			    pricechange,
			    changepercent,
			    buy,
			    sell,
			    settlement,
			    open,
			    high,
			    low,
			    volume,
			    amount,
			    ticktime,
			    per,
			    pb,
			    mktcap,
			    nmc,
			    turnoverratio
		);
	}
}

3. Test Class to run deserialization

public class IndexComponentHangqingTest {
	private Logger logger = LoggerFactory.getLogger(this.getClass());
	
	private String response;
	private byte[] res;
	private Document document;

	@Mock
	private TextDownloader textDownloader;
	
	@Before
	public void setup() throws IOException {
		MockitoAnnotations.initMocks(this);

		response = ResourceLoader.loadResource("htmltests/sina/index/hangqing_nodedata_20171219.txt").get();
		res = ResourceLoader.loadBytes("file:///H:/java/ichihedge-ws/xshell-boot/src/main/resources/htmltests/sina/index/hangqing_nodedata_20171219.txt").get();
		document = Jsoup.parse(response);
	}

	@Test
	public void test_index_parse_json() {
		//based on the result, extract the stock components.
		ObjectMapper om = new ObjectMapper();
		om.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
		//om.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));
		om.setDateFormat(new SimpleDateFormat("HH:mm:ss"));
		try{
			IndexComponentItem[] array1= om.readValue(res, IndexComponentItem[].class);
			IndexComponentItem[] array2 = om.readValue(response.getBytes(), IndexComponentItem[].class);

			List<IndexComponentItem> list1 = om.readValue(res, new TypeReference<List<IndexComponentItem>>(){});
			om.readValue(response.getBytes(), new TypeReference<List<IndexComponentItem>>(){});
			//the following line fails parsing, why?
			//om.readValue(response, new TypeReference<List<IndexComponentItem>>(){});

			for(int i=0; i<array2.length; i++){
				System.out.println(array2[i].getName());
			}
			System.out.println(array1.length);
		}catch(Exception e){
			e.printStackTrace();
		}
	}


}

Note that parsing “response” results exception (while parsing “response.getBytes()” runs totally fine):

com.fasterxml.jackson.core.JsonParseException: Unexpected character ('' (code 65279 / 0xfeff)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: [{symbol:"sh600111",code:"600111",name:"北方稀土",trade:"13.390",............
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558)

It’s obviously not because of encoding, because the test file is encoded in UTF8, the response string is read with encoding UTF8. When calling getBytes(), I didn’t even bother to specify encoding and it worked; then why string won’t work?

After checking stackoverflow and googling for half an hour, I give up for the moment. I will just stick with byte[] approach for parsing JSON for now.

After Thoughts:
a) It seems that the problem might really be caused by encoding. When Jackson parses String, underlying library might be using String.getBytes() to get byte[] for processing, which is platform dependent (if your system is with UTF8 encoding by default then it will just work, otherwise you will hit an exception). Therefore, instead of using readValue(String, …), it’s always a better practice to use readValue(byte[],…), so that you get full control on the encoding of the byte[], by String.readBytes(Charsets.UTF_8).
I had this thought when I was running the same test code in company’s workstation, when I hit exceptions, even though the test case runs fine on my home PC. To solve the problem, I had to specifically getBytes(Charsets.UTF_8).
b) Checking Jackson’s source code shows that ObjectMapper.readValue(String,Class) would invoke createJsonParser(String content) which creates IOContext, which contains encoding settings. But I can’t find where IOContext can be configured with different encoding settings.

Reference:
1 Jackson Serialization List
http://www.studytrails.com/java/json/java-jackson-Serialization-list/

How To Get Historical Data For Chinese Stock Market

This entry shows a way to obtain historical prices for securities (including individual stocks, index, and ETF) list on Shanghai Stock Exchange (SSE) and Shenzhen Stock Exchange (SZSE) via Sina Finance.

How to get historical data for 300ETF.
1. Open the link in Chrome, and launch inspect by right clicking on the page and choosing “Inspect” (or press ctrl+shift+i)
2. Under “network”, observe that the traffic is using the following to retrieve fund prices:

http://stock.finance.sina.com.cn/fundInfo/api/openapi.php/CaihuiFundInfoService.getNav?callback=jQuery1112010373892774805427_1476279920354&symbol=510300&datefrom=2016-07-01&dateto=2016-10-12&page=2

The spec of the URL is:
URL: http://stock.finance.sina.com.cn
Path: /fundInfo/api/openapi.php/CaihuiFundInfoService.getNav
Paramters:
1. callback=<callbackIdentifier>
2. symbol=<fundCode>
3. datefrom=<startDate>
4. dateto=<endDate>
5. page=<pageIndex>

*callbackIdentifer is to make sure that the result from the requesting function.
*page index is to indicate the current page. if there are 50 days of prices in total and each page will contain 20 days, then there will be 3 pages in total. 20+20+10.

how to get historical data for a stock listed in SSE and SZSE.

http://vip.stock.finance.sina.com.cn/corp/go.php/vMS_MarketHistory/stockid/002413.phtml?year=2016&jidu=2

how to get last active price of a security (stock, index, or ETF)

http://hq.sinajs.cn/format=js&func=S_Finance.upconstants.setData();&list=sz002413,s_sz395099,s_sz399106,s_sh000001,s_sz399001,s_sh000300

the result is in json format as following:

var hq_str_sz002413="雷科防务,14.050,14.130,13.940,14.120,13.880,13.930,13.940,7653866,106820947.700,40400,13.930,20800,13.920,16890,13.910,56090,13.900,46290,13.890,61153,13.940,33630,13.950,16600,13.960,14870,13.970,13900,13.980,2016-10-12,15:06:03,00";
var hq_str_s_sz395099=",0.00,0.000,0.00,182306704,26544088";
var hq_str_s_sz399106="深证综指,2047.12,3.433,0.17,182306704,26544088";
var hq_str_s_sh000001="上证指数,3058.4981,-6.7516,-0.22,1424935,15998296";
var hq_str_s_sz399001="深证成指,10788.59,6.283,0.06,182306704,26544088";
var hq_str_s_sh000300="沪深300,3300.0099,-6.5474,-0.20,686697,7238691";
S_Finance.upconstants.setData();

300ETF component last prices can be obtained by following:

http://stock.finance.sina.com.cn/hs300/api/jsonp.json/SINAFINANCE147618737228567546/Cffex_PositionsService.getConditionHS300Data?page=1&ordertype=desc&orderby=WEIGHING
http://stock.finance.sina.com.cn/hs300/api/jsonp.json/SINAFINANCE147612097896073446/Cffex_PositionsService.getConditionHS300Data?page=1&ordertype=desc&orderby=buy
http://stock.finance.sina.com.cn/hs300/api/jsonp.json/SINAFINANCE147618795626182947/Cffex_PositionsService.getConditionHS300Data?page=1&ordertype=desc&orderby=WEIGHING

Definition of the URL is as following:

1.url=http://stock.finance.sina.com.cn/hs300/api/jsonp.json
2.service=Cffex_PositionsService.getConditionHS300Data
3.data=orderby: "buy"
4.ordertype: "desc"
5.page: 1

script.type=”text/javascript”;
The function name can be defined as following (in order to generate a unique callback ID)
var fnName = “SINAFINANCE” + (+new Date()) + parseInt( Math.random() * 100000 );
var urlP = url + “/” + fnName + “/” + service + “?” + $.param(data);

Once a list of stock codes are retrieved, use the following to extract the stock prices.


http://hq.sinajs.cn/?_=1476285563683/&list=sh601318,sh600036,sh600016,sh601166,sh600000,sh600030,sz000002,sh600837,sh600519,sz000651,sh601328,sh600887,sh601288,sh601601,sh601398,sz000001,sh600104,sh601818,sh601088,sh601668,sh601169,sh601006,sh600383,sh601939,sh600015,sz000333,sh600900,sh600048,sz000858,sh601989

Reference:
1. Sina finance provides historical 300ETF prices in following link:
http://finance.sina.com.cn/fund/quotes/510300/bc.shtml
2. Historical stock prices provided by sina finance.
http://vip.stock.finance.sina.com.cn/corp/go.php/vMS_MarketHistory/stockid/002413.phtml
3. 300ETF component’s weights and their last prices.
http://finance.sina.com.cn/qizhi/hs300.html#qz

Jackson Cheatsheet

1. Jackson was expecting double-quote to start field name.
When parsing the following json text to object, parser will complain field name does not start with double-quote.

JsonParseException: Unexpected character ('d' (code 100)): was expecting double-quote to start field name

The JSON file is following:

{data:[{S:"601318",SYMBOL:"sh601318"}],date:"2014-08-29 00:00:00",num:300}

To suppress the error, enable the following configuration:

	ObjectMapper om = new ObjectMapper();
	om.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);

2. Jackson Failed To Parse Date From String

Error message is following:

org.codehaus.jackson.map.JsonMappingException: Can not construct instance of
java.util.Date from String value '2013-08-14 00:00:00': not a valid
representation.

To fix, configure customized date format for object mapping.

	ObjectMapper om = new ObjectMapper();
	om.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));

3. Json Text containing same string with different cases.
For example, a POJO parsing the following will always have error message like property conflict on SYMBOL and symbol if you have both “SYMBOL” and “symbol” as property of the POJO.

{
    S:"601318",
    SYMBOL:"sh601318", <---
    SNAME:"中国平安",
    WEIGHING:"3.77",
    INAME:"金融地产",
    HYSYMBOL:"sh000914",
    zgb:7991402541.3895,
    symbol:"sh601318", <---
    volume:"49375218",
}

The solution I could found is to make properties public, and abandon the getter/setter methods.

Example that works for parsing sina stock price

class Sina300Page{
	private List<Sina300ETFStockInfo> data;
	private Date date;
	private int num;
	public List<Sina300ETFStockInfo> getData() {
		return data;
	}
	public void setData(List<Sina300ETFStockInfo> data) {
		this.data = data;
	}
	public Date getDate() {
		return date;
	}
	public void setDate(Date date) {
		this.date = date;
	}
	public int getNum() {
		return num;
	}
	public void setNum(int num) {
		this.num = num;
	}
	
}
class Sina300ETFStockInfo{
		public String S; //:"600150",
		public String SYMBOL;//:"sh600150",
		public String symbol;//:"sh600150",
		public String SNAME;//SNAME:"中国船舶",
		public String WEIGHING; //WEIGHING:"0.31",
		public String INAME; //INAME:"工业",
		public String HYSYMBOL; //HYSYMBOL:"sh000910",
		public double ltgb; //ltgb:1426911053.5423,
		public double zgb;	//zgb:1426911053.5423,
		public String name; //name:"中国船舶",
		public String prevtrade; //prevtrade:"23.340",
		public String pricechange; //pricechange:"1.190",
		public String changepercent; //changepercent:"5.373",
		public String buy; //buy:"23.340",
		public String sell; //sell:"23.360",
		public String settlement; //settlement:"22.150",
		public String open; //open:"22.420",
		public String high; //high:"23.700",
		public String low; //low:"22.400",
		public String volume; //volume:"27162579",
		public String amount; //amount:"630928382",
		public double gxzs; //gxzs:0.537,
		public double ltsz; //ltsz:33304103989.677}
}

public class Application implements CommandLineRunner{
	private static Logger logger = LogManager.getLogger(Application.class);
	
	@Autowired
	private Environment env;
	
	@Autowired
	private EquityHistoryPersister equityHistoryPersister;
	
	@Autowired
	private EquityScraper equityScraper;
	
    public static void main(String[] args) {
    	System.setProperty("app.home", "H:/java/ichihedge-ws/commons-dao/src/main/java/com/ichihedge/emc/app");
        SpringApplication.run(Application.class, args);
    }
	public void TestSinaJson(){
		ObjectMapper om = new ObjectMapper();
		//om.configure(DeserializationFeature.USE_JAVA_ARRAY_FOR_JSON_ARRAY, false);
		om.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
		om.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));
		
		try{
			Sina300Page page = om.readValue(new File("H:\\java\\ichihedge-ws\\Documentation\\json-data.txt"), Sina300Page.class); 
			//for(int i=0; i<page.getData().size();i++){
				Sina300ETFStockInfo index = page.getData().get(0);

				logger.info(index.amount+" amount");
				logger.info(index.buy+" buy");
				logger.info(index.changepercent+" changePercent");
				logger.info(index.gxzs+" gxzs");
				logger.info(index.high+" high");
				logger.info(index.HYSYMBOL+" hysymbol");
				logger.info(index.INAME+" iname");
				logger.info(index.low+" low");
				logger.info(index.ltgb+" ltgb");
				logger.info(index.ltsz+" ltgz");
				logger.info(index.name+" name");
				logger.info(index.open+" open");
				logger.info(index.prevtrade+" prevTrader");
				logger.info(index.pricechange+" priceChange");
				logger.info(index.S+" S");
				logger.info(index.sell+" sell");
				logger.info(index.settlement+" settlement");
				logger.info(index.SNAME+" sname");
				logger.info(index.SYMBOL+" SYMBOL");
				logger.info(index.symbol+" symbol");
				logger.info(index.volume+" volume");
				logger.info(index.WEIGHING+" weighing");
				logger.info(index.zgb+" zgb");				
			//}		
		}catch(Exception e){
			e.printStackTrace();
		}
	}
}
//json-data file has following content
{data:[{S:"601318",SYMBOL:"sh601318",SNAME:"中国平安",WEIGHING:"3.77",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:4831933910.0356,zgb:7991402541.3895,symbol:"sh601318",name:"中国平安",prevtrade:"34.400",pricechange:"-0.220",changepercent:"-0.636",buy:"34.400",sell:"34.410",settlement:"34.600",open:"34.620",high:"34.750",low:"34.250",volume:"49375218",amount:"1701605543",gxzs:-0.773,ltsz:166218526505.22},{S:"600036",SYMBOL:"sh600036",SNAME:"招商银行",WEIGHING:"3.23",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:20806270598.877,zgb:25436635104.734,symbol:"sh600036",name:"招商银行",prevtrade:"18.310",pricechange:"0.030",changepercent:"0.164",buy:"18.310",sell:"18.330",settlement:"18.300",open:"18.330",high:"18.490",low:"18.260",volume:"12929420",amount:"237561461",gxzs:0.171,ltsz:380962814665.43},{S:"600016",SYMBOL:"sh600016",SNAME:"民生银行",WEIGHING:"3.14",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:27368361385.803,zgb:34369039949.365,symbol:"sh600016",name:"民生银行",prevtrade:"9.310",pricechange:"-0.020",changepercent:"-0.214",buy:"9.310",sell:"9.320",settlement:"9.330",open:"9.320",high:"9.340",low:"9.280",volume:"34307040",amount:"319143727",gxzs:-0.217,ltsz:254799444501.83},{S:"601166",SYMBOL:"sh601166",SNAME:"兴业银行",WEIGHING:"2.19",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:16434413814.402,zgb:19352373550.228,symbol:"sh601166",name:"兴业银行",prevtrade:"16.040",pricechange:"0.010",changepercent:"0.062",buy:"16.030",sell:"16.040",settlement:"16.030",open:"16.040",high:"16.060",low:"16.000",volume:"27808012",amount:"445754946",gxzs:0.044,ltsz:263607997583},{S:"600000",SYMBOL:"sh600000",SNAME:"浦发银行",WEIGHING:"1.96",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:15098151649.577,zgb:18872689561.971,symbol:"sh600000",name:"浦发银行",prevtrade:"16.580",pricechange:"0.020",changepercent:"0.121",buy:"16.560",sell:"16.580",settlement:"16.550",open:"16.570",high:"16.580",low:"16.500",volume:"12690366",amount:"209785537",gxzs:0.076,ltsz:250327354349.98},{S:"600030",SYMBOL:"sh600030",SNAME:"中信证券",WEIGHING:"1.88",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:10008780800.079,zgb:11234806113.608,symbol:"sh600030",name:"中信证券",prevtrade:"16.510",pricechange:"0.110",changepercent:"0.670",buy:"16.520",sell:"16.530",settlement:"16.410",open:"16.410",high:"16.580",low:"16.350",volume:"78154011",amount:"1285859995",gxzs:0.406,ltsz:165244971009.31},{S:"000002",SYMBOL:"sz000002",SNAME:"万科A",WEIGHING:"1.64",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:9779140507.8492,zgb:11137144936.284,symbol:"sz000002",name:"万 科Ａ",prevtrade:"26.230",pricechange:"0.260",changepercent:"1.001",buy:"26.220",sell:"26.230",settlement:"25.970",open:"25.750",high:"27.410",low:"25.560",volume:"150728616",amount:"4017712913",gxzs:0.529,ltsz:256506855520.88},{S:"600837",SYMBOL:"sh600837",SNAME:"海通证券",WEIGHING:"1.47",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:8226162338.2609,zgb:9743474325.8385,symbol:"sh600837",name:"海通证券",prevtrade:"16.100",pricechange:"0.040",changepercent:"0.249",buy:"16.090",sell:"16.100",settlement:"16.060",open:"16.070",high:"16.140",low:"16.010",volume:"13552239",amount:"217952728",gxzs:0.118,ltsz:132441213646},{S:"600519",SYMBOL:"sh600519",SNAME:"贵州茅台",WEIGHING:"1.34",INAME:"主要消费",HYSYMBOL:"sh000912",ltgb:1158174231.1242,zgb:1158174231.1242,symbol:"sh600519",name:"贵州茅台",prevtrade:"308.860",pricechange:"5.960",changepercent:"1.967",buy:"308.800",sell:"308.900",settlement:"302.960",open:"304.000",high:"309.000",low:"303.000",volume:"4173675",amount:"1279755341",gxzs:0.85,ltsz:357713693025.02},{S:"000651",SYMBOL:"sz000651",SNAME:"格力电器",WEIGHING:"1.27",INAME:"可选消费",HYSYMBOL:"sh000911",ltgb:3039932234.7249,zgb:3039932234.7249,symbol:"sz000651",name:"格力电器",prevtrade:"22.110",pricechange:"0.000",changepercent:"0.000",buy:"22.110",sell:"22.120",settlement:"22.110",open:"22.080",high:"22.220",low:"21.910",volume:"44485825",amount:"981783491",gxzs:0,ltsz:67212901709.768},{S:"601328",SYMBOL:"sh601328",SNAME:"交通银行",WEIGHING:"1.22",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:33104090222.159,zgb:75159619478.877,symbol:"sh601328",name:"交通银行",prevtrade:"5.570",pricechange:"-0.030",changepercent:"-0.536",buy:"5.570",sell:"5.580",settlement:"5.600",open:"5.590",high:"5.590",low:"5.560",volume:"45368217",amount:"252796185",gxzs:-0.211,ltsz:184389782537.43},{S:"600887",SYMBOL:"sh600887",SNAME:"伊利股份",WEIGHING:"1.21",INAME:"主要消费",HYSYMBOL:"sh000912",ltgb:3047989205.4025,zgb:3105291668.6944,symbol:"sh600887",name:"伊利股份",prevtrade:"0.000",pricechange:"0.000",changepercent:"0.000",buy:"0.000",sell:"0.000",settlement:"16.110",open:"0.000",high:"0.000",low:"0.000",volume:"0",amount:"0",gxzs:0,ltsz:0},{S:"601288",SYMBOL:"sh601288",SNAME:"农业银行",WEIGHING:"1.17",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:286521732758.8,zgb:327489503863.07,symbol:"sh601288",name:"农业银行",prevtrade:"3.140",pricechange:"-0.010",changepercent:"-0.318",buy:"3.140",sell:"3.150",settlement:"3.150",open:"3.150",high:"3.150",low:"3.140",volume:"65399176",amount:"205828949",gxzs:-0.12,ltsz:899678240862.63},{S:"601601",SYMBOL:"sh601601",SNAME:"中国太保",WEIGHING:"1.12",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:6336097852.2787,zgb:9133204819.2771,symbol:"sh601601",name:"中国太保",prevtrade:"29.850",pricechange:"0.390",changepercent:"1.322",buy:"29.850",sell:"29.920",settlement:"29.510",open:"29.590",high:"30.070",low:"29.310",volume:"7773808",amount:"231144403",gxzs:0.478,ltsz:189132520890.52},{S:"601398",SYMBOL:"sh601398",SNAME:"工商银行",WEIGHING:"1.1",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:266217129750.6,zgb:353515790838.68,symbol:"sh601398",name:"工商银行",prevtrade:"4.460",pricechange:"-0.010",changepercent:"-0.224",buy:"4.450",sell:"4.460",settlement:"4.460",open:"4.460",high:"4.460",low:"4.440",volume:"55342680",amount:"246098904",gxzs:-0.079,ltsz:1187328398687.7},{S:"000001",SYMBOL:"sz000001",SNAME:"平安银行",WEIGHING:"1.08",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:6757040333.7192,zgb:11537455326.773,symbol:"sz000001",name:"平安银行",prevtrade:"9.150",pricechange:"0.030",changepercent:"0.329",buy:"9.140",sell:"9.150",settlement:"9.120",open:"9.130",high:"9.150",low:"9.110",volume:"36337078",amount:"331886515",gxzs:0.115,ltsz:61826919053.531},{S:"600104",SYMBOL:"sh600104",SNAME:"上汽集团",WEIGHING:"1.04",INAME:"可选消费",HYSYMBOL:"sh000911",ltgb:9448925074.1926,zgb:11271910810.928,symbol:"sh600104",name:"上汽集团",prevtrade:"22.390",pricechange:"-0.080",changepercent:"-0.356",buy:"22.390",sell:"22.400",settlement:"22.470",open:"22.540",high:"22.540",low:"22.220",volume:"15734229",amount:"351921558",gxzs:-0.119,ltsz:211561432411.17},{S:"601818",SYMBOL:"sh601818",SNAME:"光大银行",WEIGHING:"0.98",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:40570099185.115,zgb:47569917423.664,symbol:"sh601818",name:"光大银行",prevtrade:"3.810",pricechange:"-0.020",changepercent:"-0.524",buy:"3.800",sell:"3.810",settlement:"3.820",open:"3.830",high:"3.830",low:"3.790",volume:"66367634",amount:"252651792",gxzs:-0.166,ltsz:154572077895.29},{S:"601088",SYMBOL:"sh601088",SNAME:"中国神华",WEIGHING:"0.92",INAME:"能源",HYSYMBOL:"sh000908",ltgb:16702034400.215,zgb:20144100447.354,symbol:"sh601088",name:"中国神华",prevtrade:"15.530",pricechange:"0.130",changepercent:"0.844",buy:"15.530",sell:"15.550",settlement:"15.410",open:"15.480",high:"15.680",low:"15.440",volume:"16777745",amount:"260859548",gxzs:0.25,ltsz:259382594235.35},{S:"601668",SYMBOL:"sh601668",SNAME:"中国建筑",WEIGHING:"0.87",INAME:"工业",HYSYMBOL:"sh000910",ltgb:30141193183.28,zgb:30289389067.524,symbol:"sh601668",name:"中国建筑",prevtrade:"6.180",pricechange:"0.010",changepercent:"0.162",buy:"6.170",sell:"6.180",settlement:"6.160",open:"6.170",high:"6.220",low:"6.140",volume:"79311038",amount:"489585637",gxzs:0.045,ltsz:186272573872.67},{S:"601169",SYMBOL:"sh601169",SNAME:"北京银行",WEIGHING:"0.86",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:9155036374.5084,zgb:10780808259.96,symbol:"sh601169",name:"北京银行",prevtrade:"9.100",pricechange:"-0.020",changepercent:"-0.219",buy:"9.100",sell:"9.110",settlement:"9.120",open:"9.130",high:"9.150",low:"9.080",volume:"29878491",amount:"272018944",gxzs:-0.061,ltsz:83310831008.026},{S:"601006",SYMBOL:"sh601006",SNAME:"大秦铁路",WEIGHING:"0.82",INAME:"工业",HYSYMBOL:"sh000910",ltgb:15091118934.623,zgb:15091118934.623,symbol:"sh601006",name:"大秦铁路",prevtrade:"6.500",pricechange:"0.000",changepercent:"0.000",buy:"6.490",sell:"6.500",settlement:"6.490",open:"6.530",high:"6.540",low:"6.460",volume:"44133843",amount:"286924439",gxzs:0,ltsz:98092273075.048},{S:"600383",SYMBOL:"sh600383",SNAME:"金地集团",WEIGHING:"0.75",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:4471508572,zgb:4471508572,symbol:"sh600383",name:"金地集团",prevtrade:"12.000",pricechange:"0.000",changepercent:"0.000",buy:"11.990",sell:"12.000",settlement:"12.000",open:"11.960",high:"12.150",low:"11.870",volume:"15030775",amount:"180630083",gxzs:0,ltsz:53658102864},{S:"601939",SYMBOL:"sh601939",SNAME:"建设银行",WEIGHING:"0.71",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:9641625894.03,zgb:251261032373.43,symbol:"sh601939",name:"建设银行",prevtrade:"5.200",pricechange:"0.000",changepercent:"0.000",buy:"5.200",sell:"5.210",settlement:"5.210",open:"5.220",high:"5.220",low:"5.190",volume:"48047380",amount:"250199358",gxzs:0,ltsz:50136454648.956},{S:"600015",SYMBOL:"sh600015",SNAME:"华夏银行",WEIGHING:"0.7",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:6589301182.7386,zgb:9044113829.0205,symbol:"sh600015",name:"华夏银行",prevtrade:"10.200",pricechange:"0.050",changepercent:"0.493",buy:"10.190",sell:"10.200",settlement:"10.150",open:"10.170",high:"10.210",low:"10.120",volume:"40284336",amount:"409836661",gxzs:0.111,ltsz:67210872063.934},{S:"000333",SYMBOL:"sz000333",SNAME:"美的集团",WEIGHING:"0.69",INAME:"可选消费",HYSYMBOL:"sh000911",ltgb:1731537572.6606,zgb:4254485613.945,symbol:"sz000333",name:"美的集团",prevtrade:"27.000",pricechange:"0.050",changepercent:"0.186",buy:"26.990",sell:"27.000",settlement:"26.950",open:"26.900",high:"27.140",low:"26.680",volume:"22004789",amount:"592389006",gxzs:0.041,ltsz:46751514461.836},{S:"600900",SYMBOL:"sh600900",SNAME:"长江电力",WEIGHING:"0.68",INAME:"公用事业",HYSYMBOL:"sh000917",ltgb:10126536126.36,zgb:17144351464.435,symbol:"sh600900",name:"长江电力",prevtrade:"13.480",pricechange:"0.130",changepercent:"0.974",buy:"13.470",sell:"13.480",settlement:"13.350",open:"13.310",high:"13.540",low:"13.290",volume:"17214016",amount:"231111950",gxzs:0.214,ltsz:136505706983.33},{S:"600048",SYMBOL:"sh600048",SNAME:"保利地产",WEIGHING:"0.67",INAME:"金融地产",HYSYMBOL:"sh000914",ltgb:10763943669.91,zgb:10763943669.91,symbol:"sh600048",name:"保利地产",prevtrade:"9.250",pricechange:"-0.010",changepercent:"-0.108",buy:"9.250",sell:"9.260",settlement:"9.260",open:"9.190",high:"9.380",low:"9.180",volume:"53655557",amount:"497446319",gxzs:-0.023,ltsz:99566478946.664},{S:"000858",SYMBOL:"sz000858",SNAME:"五粮液",WEIGHING:"0.66",INAME:"主要消费",HYSYMBOL:"sh000912",ltgb:3870356922.4605,zgb:3870356922.4605,symbol:"sz000858",name:"五 粮 液",prevtrade:"33.470",pricechange:"-0.200",changepercent:"-0.594",buy:"33.470",sell:"33.480",settlement:"33.670",open:"33.770",high:"33.870",low:"33.210",volume:"22611729",amount:"757048019",gxzs:-0.126,ltsz:129540846194.75},{S:"601989",SYMBOL:"sh601989",SNAME:"中国重工",WEIGHING:"0.66",INAME:"工业",HYSYMBOL:"sh000910",ltgb:16117342215.454,zgb:18215161712.813,symbol:"sh601989",name:"中国重工",prevtrade:"6.650",pricechange:"0.320",changepercent:"5.047",buy:"6.650",sell:"6.660",settlement:"6.340",open:"6.410",high:"6.730",low:"6.400",volume:"163694073",amount:"1083836741",gxzs:1.074,ltsz:107180325732.77}],date:"2014-08-29 00:00:00",num:300}

Reference:
1. Parsing Error On Json Attribute
http://www.mkyong.com/java/jackson-was-expecting-double-quote-to-start-field-name/
2. Example Code To Get Json Working
http://www.mkyong.com/java/how-to-convert-java-object-to-from-json-jackson/

Correctly Configuring MappingJackson2MessageConverter For Messaging Via JSON

Apparently, neither the API document nor any tutorial is covering the proper configuration of the message converter for MappingJackson2MessageConverter. After hours of trying and googling, I still receive loads of exceptions (like NullPointException, ClassNotFoundException, and etc) and nothing was ideal.

After reading the source code carefully, I figured out how this converter would work properly if you configure it correctly.

Two most import method of MessageConverter is toMessage() and fromMessage(), where the formal method serializes an object into a JMS message, and the latter deserializes a message into an object.

In MappingJackson2MessageConverter, the message is JSON text, so how would the message converter know which class to deserialize the text into? The magic is hidden and buried in the source code (Spring should have documented the mechanism properly so that people like me should not have wasted a day on this).

JMS message has a head section which contains properties (such as destination, object type, and anything that you want to add), and a payload section which is the message body (in this case, a JSON text). The MappingJackson2MessageConverter.convertAndSend() will fill in a property containing class type information of the JSON text into the message header right before the message gets send; and likewise, receiveAndConvert() will try to extract the class type information from the message header in order to deserialize it into a correct typed object.

Now you’ve understood the undocumented magic! The following is exactly how everything should happen in orderly fashion:

1. configure the type-id-property-name by setTypeIdPropertyName().
This can be any name, and it has nothing to do with the class type information. However, it’s important that both sender and receiver use the same name if they are use different message converter instances. Nothing to worry if you are using the singleton object.

messageConverter.setTypeIdPropertyName("ichihedge.queue.json.classname");

2. configure the Type-Id mappings by setTypeIdMappings().
All beans should be defined in the mapping, in class type of Map. For senders, it doesn’t matter too much; but for receivers, this mapping information is very important, because when deserializing json, the receiver needs to know which class type to deserialize into.

Map<String,Class> typeIdMappings = new Map<String,Class>();
typeIdMappings.put(ActionBean.class.getName(), ActionBean.class);
messageConverter.setTypeIdMappings(typeIdMappings);

3. fire away by simply calling jmsTemplate.convertAndSend(…) and jmsTemplate.receiveAndConvert(…)

You may ignore the point 4, as it’s some draft that I kept when analyzing the source code of MappingJackson2MessageConverter.

4 behind the scene
4.1 when MappingJackson2MessageConverter calls toMessage(), right before returning the message object, it will search and set the type information of the message object, via pseudo code below:

mc.setTypeIdOnMessage(object,message);
typeIdPropertyName="ichihedge.queue.json.classname"
typeId=classIdMappings.get("hello.jms.ActionBean");
message.setStringProperty(ActionBean.class.getName(), ActionBean.class);

Likewise, when MappingJackson2MessageConverter calls fromMessage(), it will look up for class type information in order to do deserialization as below:

getJavaTypeForMessage(message);
typeId=message.getStringProperty("ichihedge.queue.json.classname");
mappedCalss=this.idClassMappings.get(typeId);
...//then converts json to object of mappedClass

5. Code listing for reference
This section lists the most critical part to enable the json serialization and deserialization – the configuration on message converter. Rest of the code is same as before (refer to other entry for a detailed example on how to send a message via JmsTemplate). 

	@Bean
	public MessageConverter messageConverter(ObjectMapper objectMapper){
		MappingJackson2MessageConverter messageConverter = new MappingJackson2MessageConverter();
		messageConverter.setObjectMapper(objectMapper);
		messageConverter.setTargetType(MessageType.TEXT);
		

		messageConverter.setTypeIdPropertyName("ichihedge.queue.json.classname");
		//now set idMappings for serialization/deserialization
		HashMap<String, Class<?>> idMapping = new HashMap<String, Class<?>>();
		idMapping.put(ActionBean.class.getName(), ActionBean.class);
		messageConverter.setTypeIdMappings(idMapping);
		
		return messageConverter;
	}

Sender that sends out two messages of different types.

		jmsOperations.convertAndSend(queue, new HelloBean("Mike", " hello how are you?", new Date(System.currentTimeMillis())));
		jmsOperations.convertAndSend(queue, new ActionBean("Jackson", 50.12345));

Receiver that receives two messages and dump the information.

		logger.info(jmsOperations.receiveAndConvert(queue));
		logger.info(jmsOperations.receiveAndConvert(queue));

JConsole shows the results after sender sends out two messages, note on the String Property.

Capture

Receiver results from console:

Capture.JPG

Reference
1. MappingJackson2MessageConverter.java Source Code
http://grepcode.com/file/repository.springsource.com/org.springframework/org.springframework.jms/3.1.4/org/springframework/jms/support/converter/MappingJackson2MessageConverter.java

2. JmsTemplate.java Source Code
http://grepcode.com/file/repo1.maven.org/maven2/org.springframework/spring-jms/4.2.0.RELEASE/org/springframework/jms/core/JmsTemplate.java#JmsTemplate

3. Others discussing similar issues:
http://stackoverflow.com/questions/32385062/spring-mappingjackson2messageconverter-give-null-pointer-exception