Howto Use Application Properties in Spring

This entry shows where to put the application.properties to be automatically picked up by spring-boot, and how to access the properties defined in the file.

According to reference manual on spring’s official website, the “application.properties” will be added to “Environment” automatically if you put it under the right folder.

SpringApplication will load properties from application.properties files in the following locations and add them to the Spring Environment:

  1. A /config subdirectory of the current directory.
  2. The current directory
  3. A classpath /config package
  4. The classpath root

This entry shows an explicit way to load the properties file, by using “PropertySource” annotation.

To achieve the same goal, we can simple create a “config” folder and place “application.properties” under the folder. Spring context will load those settings into Environment.

As an experiment, I tried to create properties in the following 3 locations:

  1. /config under “src/main/java”, which will create “/config” in the classpath.
  2. root directory, which will create a properties file under the current working directory.
  3. command line argument.

The following screenshot shows the folder structure, and the running result of the property precedence.

properties

The following screenshot shows the command line properties defined in Eclipse.

properties

1) /config/application.properties


disc.title=Rattle and Hum (/config)
disc.artist=U2 (/config)
disc.price=$20.1 (/config)

2) (current working directory)/application.properties

disc.title=Rattle and Hum (current working directory)
disc.artist=U2 (current working directory)
#disc.price=$20 (current working directory)

3) AppConfig.java, which does not use PropertySource.

package hello;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.context.support.PropertySourcesPlaceholderConfigurer;
import org.springframework.core.env.Environment;

@Configuration
public class AppConfig {
	@Autowired
	Environment env;

	@Bean
	public BlankDisc disc(){
		BlankDisc disc = new BlankDisc();
		disc.setArtist(env.getProperty("disc.artist"));
		disc.setTitle(env.getProperty("disc.title"));
		disc.setPrice(env.getProperty("disc.price"));
		return disc;
	}

	@Bean
	public static PropertySourcesPlaceholderConfigurer properties(){
		PropertySourcesPlaceholderConfigurer conf = new PropertySourcesPlaceholderConfigurer();
		return conf;
	}
}

4) MainApp.java

package hello;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.core.env.Environment;

@Import(AppConfig.class)
public class Application {
	private static Logger logger = LogManager.getLogger(Application.class);

	@Autowired
	private Environment env;

    public static void main(String[] args) throws Exception {
        ApplicationContext ctx = SpringApplication.run(Application.class);

        BlankDisc disc = ctx.getBean(BlankDisc.class);
        logger.info(disc);
    }
}

5) BlankDisc.java, the model that takes values from properties file.

package hello;

public class BlankDisc {
	private String title;
	private String artist;
	private String price;

	public String getTitle() {
		return title;
	}
	public void setTitle(String title) {
		this.title = title;
	}
	public String getArtist() {
		return artist;
	}
	public void setArtist(String artist) {
		this.artist = artist;
	}

	public String getPrice() {
		return price;
	}
	public void setPrice(String price) {
		this.price = price;
	}
	@Override
	public String toString(){
		return "{title="+getTitle()+", artist="+getArtist()+", " +getPrice()+ "}";
	}
}

6) Property file can be loaded via classpath instead of the file system path.
Put “application.properties” under /src/main/resources, and reference the property file in configuration class via @PropertySource(“classpath:application.properties”)

Reference:
1. http://docs.spring.io/spring-boot/docs/current/reference/html/howto-properties-and-configuration.html
Under section 69.2.
2. http://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html
Under section 24.3 for “Application property files”.

Spring Data Support For MongoDB

In general, spring data has to follow the following main steps when working with database.
1) configure/setup database connection; usually involves settings such as: server, port, username, password, and etc.
2) query database; usually via plan queries, or spring auto-generated classes, or some other mechanisms.
3) object data mapping definitions, which will serialize the java objects into database entries, and deserialize data entries back into java objects.

In particular, when spring data works with MongoDB, it follows the three main steps as well.
1) Configure Spring Data for MongoDB
2) Query MongoDB With Spring Data
3) Data Object Mapping In Spring Data For MongoDB

Reference
http://docs.spring.io/spring-data/mongodb/docs/current/reference/html/

Data Object Mapping In Spring Data For MongoDB

Data models play an important part when spring data works with MongoDB. They are POJO, which maps to JSON in MongoDB documents.

package hello.model;

import java.util.Collection;
import java.util.LinkedHashSet;

import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
import org.springframework.data.mongodb.core.mapping.Field;

/**
 * Object model that maps to MongoDB
 * @author Wofong
 *
 */
@Document
public class Order {
	@Id
	private String id;
	
	@Field("client")
	private String customer;
	
	private String type;
	
	private Collection<Item> items = new LinkedHashSet<Item>();

	public String getId() {
		return id;
	}

	public String getCustomer() {
		return customer;
	}

	public String getType() {
		return type;
	}

	public Collection<Item> getItems() {
		return items;
	}

	public void setCustomer(String customer) {
		this.customer = customer;
	}

	public void setType(String type) {
		this.type = type;
	}
	
	@Override
	public String toString(){
		return "{id="+id+", customer="+customer+", type="+type+"}";
	}
}

Query MongoDB With Spring Data

This entry explains two ways to query Mongo DB via Spring Data

1) Using MongoTemplate
After spring-data has configured mongo-db access, we will have one MongoTemplate bean in the context. MongoTemplate (available via bean-wiring) provides a full list of queries which can get we started on querying MongoDB.

If you need a more complicated queries, please consider to use the Query class.

package hello.main;

import static org.springframework.data.mongodb.core.query.Criteria.where;
import static org.springframework.data.mongodb.core.query.Query.query;

import java.util.Iterator;
import java.util.List;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.context.annotation.Import;
import org.springframework.data.mongodb.core.MongoOperations;

import hello.model.Item;
import hello.model.Order;

@Import(MongoConfigDirectly.class) 
//@Import(MongoConfigSpring.class)
public class AppMongoTemplate implements CommandLineRunner{
	private static Logger logger = LogManager.getLogger(AppMongoTemplate.class);

	@Autowired
	private MongoOperations mongo;

	@Override
	public void run(String... args) throws Exception {
		//display information on mongo-db configuration
		logger.info("mongo configuration " + mongo.toString());
		
		//get the count of orders
		long orderCount = mongo.getCollection("order").count();
		logger.info("count(): " + orderCount);
		
		//save a new order
		Order order = new Order();
		order.setCustomer("Huafeng");
		order.setType("Broker Fee");
		//mongo.save(order, "order");
		mongo.save(order);
		
		order = new Order();
		order.setCustomer("Mike");
		order.setType("Taobao");
		order.getItems().add(new Item("Shaver", 20.1d, 1));
		mongo.insert(order);
		
		//find one order
		String id="";
		Order findOneOrder = mongo.findById(id, Order.class);
		
		//find a list of orders
		List<Order> findListOrders = mongo.find(query(where("client").is("Huafeng")), Order.class);
		for(Iterator<Order> i=findListOrders.iterator();i.hasNext();){
			logger.info(i.next());
		}
		//mongo.dropCollection("order");
	}
}

2) Using repositories support from Spring Data
Spring Data support for MongoDB has similar feature for JPA. One can extend the repository interface to have Spring Data support. Basic queries are straight from out of the box.

More complex queries can be customized via the method signature, or via query annotation.

package hello.main;

import java.util.Iterator;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.context.annotation.Import;

import hello.model.Order;
import hello.repositories.OrderRepository;

//@Import(MongoConfigDirectly.class) 
@Import(MongoConfigSpring.class)
public class AppMongoRepository implements CommandLineRunner {
	private static Logger logger = LogManager.getLogger(AppMongoRepository.class);
	
	@Autowired
	private OrderRepository repository;
	
	@Override
	public void run(String... args) throws Exception {
		logger.info("count: "+repository.count());
		for(Iterator<Order> i =repository.findAll().iterator();i.hasNext();){
			logger.info(i.next());
		}
	}

}
package hello.repositories;

import java.util.List;

import org.springframework.data.mongodb.repository.MongoRepository;

import hello.model.Order;

public interface OrderRepository extends MongoRepository<Order, String> {
	List<Order> findByCustomer(String c);
	List<Order> findByCustomerLike(String c);
	List<Order> findByCustomerAndType(String c, String t);
	List<Order> findByCustomerLikeAndType(String c, String t);
}

Configure Spring Data For MongoDB

,Spring Data needs to be configured for MongoDB connection settings before it can be useful, the settings involved are: host name, port number, (user name, password,) and the target database name of interest.

There’re different ways to configure Spring Data to enable MongoDB support. This entry presents two ways of configuration. Either way works fine.

1) Configure directly for MongoTemplate;

This is a direct approach, and you could easily tell from the code on what it is trying to achieve. The mongoTemplate bean can be used directly in spring application via bean-wiring; alternatively, if you are using repository support from Spring, mongoTemplate will be invoked under the cover.

package hello.main;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoFactoryBean;
import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;

import com.mongodb.Mongo;

/**
 * Configuration that defines two beans as required to enable Spring-Data for MongoDB
 * @author Wofong
 *
 */
@Configuration
public class MongoConfigDirectly {
	@Bean
	public MongoFactoryBean mongo(){
		MongoFactoryBean mongo = new MongoFactoryBean();
		mongo.setHost("localhost");
		return mongo;
	}
	@Bean
	public MongoOperations mongoTemplate(Mongo mongo){
		return new MongoTemplate(mongo, "OrdersDB");
	}
}

2) Configure by extending AbstractMongoConfiguration

The code is cleaner; less readable than the previous approach.

package hello.main;

import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.config.AbstractMongoConfiguration;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;

import com.mongodb.Mongo;
import com.mongodb.MongoClient;

@Configuration
@EnableMongoRepositories("hello.repositories")
public class MongoConfigSpring extends AbstractMongoConfiguration{

	@Override
	protected String getDatabaseName() {
		return "OrdersDB";
	}

	@Override
	public Mongo mongo() throws Exception {
		MongoClient mongo = new MongoClient("localhost"); //MongoClient accepts host and port.
		return mongo;
	}
}

Using PropertySource To Work With Properties

This entry shows an example on how to read configuration from a “*.properties” file, and make use of the configuration in beans.

1. app.properties (this is the target property file, and it is packaged in jar file.)

disc.title=Rattle and Hum
disc.artist=U2
disc.price=$20
disc.two.title=Juzi Xiangshui
disc.two.artist=Ren Xian Qi
disc.two.price=$19.90

2. BlankDisc.java (a data model)

package hello;

public class BlankDisc {
	private String title;
	private String artist;
	public String getTitle() {
		return title;
	}
	public void setTitle(String title) {
		this.title = title;
	}
	public String getArtist() {
		return artist;
	}
	public void setArtist(String artist) {
		this.artist = artist;
	}
	@Override
	public String toString(){
		return "{title="+getTitle()+", artist="+getArtist()+"}";
	}
}

3. AppConfig.java (spring config file)

package hello;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.context.support.PropertySourcesPlaceholderConfigurer;
import org.springframework.core.env.Environment;

@Configuration
@PropertySource("classpath:/hello/app.properties")
public class AppConfig {
	@Autowired
	Environment env;

	@Bean(name="DiscOne")
	@Primary
	public BlankDisc disc(){
		BlankDisc disc = new BlankDisc();
		disc.setArtist(env.getProperty("disc.artist"));
		disc.setTitle(env.getProperty("disc.title"));
		disc.setPrice(env.getProperty("disc.price"));
		return disc;
	}

	@Value("#{environment['disc.two.title']}")
	private String title;
	@Value("${disc.two.artist}")	
	private String artist;
	@Value("${disc.two.price}")
	private String price;
	@Bean(name="DiscTwo")
	public BlankDisc discTwo(){
		BlankDisc disc = new BlankDisc();
		disc.setArtist(artist);
		disc.setTitle(title);
		disc.setPrice(price);
		return disc;
	}

	@Bean
	public static PropertySourcesPlaceholderConfigurer properties(){
		PropertySourcesPlaceholderConfigurer conf = new PropertySourcesPlaceholderConfigurer();
		//conf.setLocation(new ClassPathResource("classpath:app.properties"));
		return conf;
	}
}

There are different ways to wire the properties.
3.1 Using SpEL for binding, e.g. @Value(“#{environment[‘disc.two.title’]}”)
This will use SpEL to evaluate the expression which reads a property from Environment.
3.2 Using Property binding, e.g. @Value(“${disc.two.artist}”)
This will use PropertySourcesPlaceholderConfigurer to assign the properties. It requires the bean to be defined.

4. Application.java (the main program)

package hello;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.core.env.Environment;

@Import(AppConfig.class)
public class Application {
	private static Logger logger = LogManager.getLogger(Application.class);

	@Autowired
	private Environment env;

    public static void main(String[] args) throws Exception {
        ApplicationContext ctx = SpringApplication.run(Application.class);

        BlankDisc disc = ctx.getBean(BlankDisc.class);
        logger.info(disc);

        logger.info(ctx.getBean("DiscTwo"));
    }
}

Reference:
1) http://www.concretepage.com/spring/example_propertysource_spring
2) http://www.blog.jamesdbloom.com/UsingPropertySourceAndEnvironment.html
3) http://www.mkyong.com/spring/spring-propertysources-example/

沪深300指数

S

The index contains the most liquid-traded stocks in Shanghai and Shenzhen stock exchanges. The stocks tracked under this index has blue chips (those from 50ETF) and good quality medium to large capital stocks. It has around 60% of the market value of the entire share market, ranging from various sectors in China, so it represents A shares very well.

The component stocks in this index will be good candidates for stock selection, and the benchmark is traded under 300ETF (510300).

Reference:
1) http://finance.sina.com.cn/qizhi/hs300.html#qz (a good place to get daily prices for 300 composition stocks of 300ETF)
2) http://www.csindex.com.cn/sseportal/csiportal/zs/jbxx/report.do?code=000300&subdir=1

PollingConsumer With Spring Integration

Spring framework has built Spring-Integration on top of Spring JMS. It was quite confusing to me at first, as I wanted to find a way to get JMS/ActiveMQ working, but the reference and guide has both JMS and Integration. Anyway after couple of days reading and trying, I managed to get the messaging working using channels. There could be better ways to make it work, which I would probably need more time before I could arrive to a working example.

What I need is a polling consumer. The producer will generate a loads of task items, and send them to the channel, all within a short time frame; then the consumer will take one task item at one time to work on it and the consumer will have plenty time before consuming another task item from the channel.

There are three classes involved to show how to achieve this: 1) Main, 2) Sender 3) Receiver.

package com.wf.playground;

import javax.jms.ConnectionFactory;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.channel.QueueChannel;
import org.springframework.integration.core.MessagingTemplate;
import org.springframework.jms.annotation.EnableJms;
import org.springframework.jms.config.JmsListenerContainerFactory;
import org.springframework.jms.config.SimpleJmsListenerContainerFactory;
import org.springframework.messaging.PollableChannel;
import org.springframework.scheduling.annotation.EnableScheduling;

@SpringBootApplication
@EnableJms
@EnableScheduling	
public class Application {
	private static Logger logger = LogManager.getLogger(Application.class);

   
    //a pollable-channel with size of 1000
    @Bean
    PollableChannel myPollableChannel(){
    	return new QueueChannel(1000);
    }
    
    //prepare a messaging-template, with timeout (send/receive) as 0
    //and the default channel to myPollableChannel.
    @Bean
    MessagingTemplate myMessagingTemplate(){
		MessagingTemplate template = new MessagingTemplate();
		template.setSendTimeout(0);
		template.setReceiveTimeout(0);
		template.setDefaultChannel(myPollableChannel());
		return template;
    }
    
	public static void main(String[] args){
		//ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
		SpringApplication.run(Application.class, args);
	}
}

 

package com.wf.playground;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.integration.core.MessagingTemplate;
import org.springframework.integration.endpoint.PollingConsumer;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.MessagingException;
import org.springframework.messaging.PollableChannel;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
@EnableScheduling
public class MyScheduledSIPoller {
	private static Logger logger = LogManager.getLogger(MyScheduledSIPoller.class);
	@Autowired
	private PollableChannel myPollableChannel;
	@Autowired
	private MessagingTemplate myMessagingTemplate;
	
	@Bean
	PollingConsumer myPollingConsumer(){
		MessageHandler mh = new MessageHandler(){
			@Override
			public void handleMessage(Message<?> message) throws MessagingException {
				// TODO Auto-generated method stub
				//logger.info("handleMessage(): " + message.toString());
			}
		};
		
		PollingConsumer consumer = new PollingConsumer(myPollableChannel, mh);
		consumer.setMaxMessagesPerPoll(1);
		consumer.setReceiveTimeout(10);

		return consumer;
	}    

	@Scheduled(fixedRate=1000)
	public void poll(){
		Message<?> msg = myMessagingTemplate.receive();
		
		if(msg!=null){
			logger.info("received msg: "+msg);
		}else{
			logger.warn("consumer polled an empty message.");
		}
	}
	
}

 

package com.wf.playground;

import java.text.SimpleDateFormat;
import java.util.Date;

import org.apache.log4j.LogManager;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.integration.core.MessagingTemplate;
import org.springframework.messaging.Message;
import org.springframework.messaging.PollableChannel;
import org.springframework.messaging.support.GenericMessage;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
@EnableScheduling
public class MyScheduledSISender {
	private static Logger logger = LogManager.getLogger(MyScheduledSISender.class);
	private static final SimpleDateFormat sdf = new SimpleDateFormat("HH:mm:ss");

	@Autowired
	private PollableChannel myPollableChannel;
	
	@Autowired
	private MessagingTemplate myMessagingTemplate;
	
	@Scheduled(fixedRate=5000)
	public void reportCurrentTime(){
		
		String content = "test message from sender @"+sdf.format(new Date(System.currentTimeMillis()));
		//create the message to be sent
		Message<?> msg = new GenericMessage<String>(content);
		
		logger.info("sending content="+content);
		myMessagingTemplate.send(myPollableChannel, msg);
	}
}


screenshot.jpg

A Scheduled Messaging Example Using Spring

Spring has made lots of things simpler, since it has built handy tools in the box, therefore, one only needs to write POJO and wire things up. Here’s how to create a scheduled messaging with Spring’s scheduling/jms support.

@SpringBootApplication
@EnableJms
@EnableScheduling
public class Application {

    @Bean
    JmsListenerContainerFactory<?> myJmsContainerFactory(ConnectionFactory connectionFactory) {
        SimpleJmsListenerContainerFactory factory = new SimpleJmsListenerContainerFactory();
        factory.setConnectionFactory(connectionFactory);
        return factory;
    }

	public static void main(String[] args){
		//ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
		SpringApplication.run(Application.class, args);
	}
}
@Component
public class MyReceiver {
	@Autowired
	ConfigurableApplicationContext context;

	@JmsListener(destination="mailbox-destination", containerFactory = "myJmsContainerFactory")
	public void receiveMessage(String message){
		System.out.println("Received <"+ message + ">");
	}
}

@Component
public class MyScheduledSender {
	private static Logger logger = LogManager.getLogger(MyScheduledSender.class);
	private static final SimpleDateFormat dateFormat = new SimpleDateFormat("HH:mm:ss");

	@Autowired
	ConfigurableApplicationContext context;

	@Scheduled(fixedRate=5000)
	public void reportCurrentTime(){
		logger.info("Current time is now " + dateFormat.format(new Date()));
		//send a message
		MessageCreator messageCreator = new MessageCreator(){
			@Override
			public Message createMessage(Session session) throws JMSException{
				return session.createTextMessage(String.format("{timestamp=%s}", dateFormat.format(new Date(System.currentTimeMillis()))));
			}
		};
		JmsTemplate jmsTemplate = context.getBean(JmsTemplate.class);
		logger.info("sending a new message...");
		jmsTemplate.send("mailbox-destination", messageCreator);
	}
}

How To Extract Stock Prices From Sina Finace

The target website in question is here: http://money.finance.sina.com.cn/corp/go.php/vMS_MarketHistory/stockid/600048.phtml

The aim is to extract the historical prices from the table in the web page.

One way to work with it is by using regexp to parse the HTML code, which was exactly what I have done. But later, I realized that there’s a much easier way to parse the HTML code.

Jsoup is the tool in choice. The following shows the steps on how to do data scrapping via Jsoup.

The view-source of the webpage looks like the following.

jsoup.JPG

The table in our interest is named “FundHoldSharesTable”.

1) Download the html document from the URL.

Document doc = Jsoup.connect(URL).get();

2) Perform a jQuery with select function, and aim for the right table in the html.

Element table = doc.select(&amp;quot;table[id=FundHoldSharesTable]&amp;quot;).first();

3) With the table element given, the rest is quite straightforward.

			Elements rows = table.select(&amp;quot;tr:not(thead&amp;gt;tr)&amp;quot;);

			int index=0;
			row=rows.get(index++);
			System.out.println(row.text());

			while(index&amp;lt;rows.size()){
				System.out.print(index+&amp;quot;: &amp;quot;);
				row=rows.get(index++);
				for(Iterator&amp;lt;Element&amp;gt; i=row.select(&amp;quot;td&amp;quot;).iterator();i.hasNext();){
					System.out.print(i.next().text() + &amp;quot;, &amp;quot;);
				}
				System.out.println();
			}			

4) The result looks like this:

jsoup.JPG

It might be tricky to work with jQuery initially, but if you spend couple of minutes to play with the selectors, Jsoup will give you nice results.

Reference:

1. Jsoup Selector Syntax: http://jsoup.org/cookbook/extracting-data/selector-syntax