CURL in Windows

To post json in windows, CURL needs to have special treatment because single quote does not work.

1 JSON via command line 1.1 JSON quoted with escape character

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{\"name\": \"value\"}" http://localhost:8080/process

1.2 JSON quoted with another double quote

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d "{""name"": ""value""}" http://localhost:8080/process

2 JSON via text file

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d @json.txt http://localhost:8080/process

On the same directory, create a file “json.txt” with following content.

{"name": "value"}

Reference: https://stackoverflow.com/questions/11834238/curl-post-command-line-on-windows-restful-service

Securing Rest Service

I have scrapers running in cloud to download stock prices, and then how do I expose the service to my home pc for read only access? Spring Data provides a toolbox for it by using JPA and Rest Services. One thing to note is on data security.

Create project from “start.spring.io”, and include: web, jpa and security as the core modules; and add additional dependencies for sqlite.

<dependency>
	<groupId>org.xerial</groupId>
	<artifactId>sqlite-jdbc</artifactId>
	<version>3.42.0.0</version>
</dependency>
<dependency>
	<groupId>org.hibernate.orm</groupId>
	<artifactId>hibernate-community-dialects</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/com.google.guava/guava -->
<dependency>
	<groupId>com.google.guava</groupId>
	<artifactId>guava</artifactId>
	<version>32.1.2-jre</version>
</dependency>
spring.datasource.initialization-mode=always
spring.jpa.generate-ddl=true
spring.jpa.database-platform=org.hibernate.community.dialect.SQLiteDialect
spring.datasource.platform=sqlite3
spring.datasource.driver-class-name = org.sqlite.JDBC
spring.datasource.url = jdbc:sqlite:C:/scratch/hello_boot/sqlitesample.db
#spring.datasource.url = jdbc:sqlite:test
spring.datasource.dbcp2.default-auto-commit=true
spring.datasource.username = 
spring.datasource.password = 

Add a controller to provide static contents, and later on you can replace it to call from Jpa services.

@RestController
public class ResourceController {
	@GetMapping("/home")
	public String homeEndpoint() {
		return "Hello Security!";
	}
}

Call service from curl, and notice the error on authentication. screenshot to be created.

Exclude default autoconfiguration for security, by either applicatoin.properties or from annotation @SpringBootApplication.

@SpringBootApplication(
	exclude= {
		SecurityAutoConfiguration.class,
		UserDetailsServiceAutoConfiguration.class
	}
)
public class HellosecurityApplication {

	public static void main(String[] args) {
		SpringApplication.run(HellosecurityApplication.class, args);
	}
}
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.security.SecurityAutoConfiguration

Add authentication for rest services by api-token, by building filter, authentication-service, and authentication object.

public class AuthenticationFilter extends GenericFilterBean{

	@Override
	public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
			throws IOException, ServletException {
		try {
			Authentication authentication = AuthenticationService.getAuthentication((HttpServletRequest)request);
			SecurityContextHolder.getContext().setAuthentication(authentication);
		}catch(Exception e) {
			HttpServletResponse httpResponse = (HttpServletResponse) response;
httpResponse.setStatus(HttpServletResponse.SC_UNAUTHORIZED);			httpResponse.setContentType(MediaType.APPLICATION_JSON_VALUE);
			PrintWriter writer = httpResponse.getWriter();
			writer.print(e.getMessage());
			writer.flush();
			writer.close();
		}
		chain.doFilter(request, response);
	}
}
public class AuthenticationService {
	private static final String AUTH_TOKEN_HEADER_NAME="X-API-KEY";
	private static final String AUTH_TOKEN = "mytoken12345678";
	
	public static Authentication getAuthentication(HttpServletRequest request) {
		String apiKey = request.getHeader(AUTH_TOKEN_HEADER_NAME);
		if(apiKey==null || !apiKey.equals(AUTH_TOKEN)) {
			throw new BadCredentialsException("Invalid API Key");
		}
		
		return new ApiKeyAuthentication(apiKey, AuthorityUtils.NO_AUTHORITIES);
	}
}
public class ApiKeyAuthentication extends AbstractAuthenticationToken {
	private final String apiKey;
	public ApiKeyAuthentication(String apiKey, Collection<? extends GrantedAuthority> authorities) {
		super(authorities);
		this.apiKey = apiKey;
		setAuthenticated(true);
	}

	@Override
	public Object getCredentials() {
		return null;
	}
	
	@Override
	public Object getPrincipal() {
		return apiKey;
	}
}

Enable SecurityFilterChain in config

@Configuration
@EnableWebSecurity
public class SecurityConfig {

	@Bean
	public SecurityFilterChain filterChain(HttpSecurity http) throws Exception{
		http.csrf()
		.disable()
		.authorizeHttpRequests()
		.requestMatchers("/**")
		.authenticated()
		.and()
		.httpBasic()
		.and()
		.sessionManagement()
		.sessionCreationPolicy(SessionCreationPolicy.STATELESS)
		.and()
		.addFilterBefore(
			new AuthenticationFilter(),
			UsernamePasswordAuthenticationFilter.class
		);
		return http.build();
	}
}

Test curl on rest service curl –location –request GET “http://localhost:8080/home” curl –location –request GET “http://localhost:8080/home” –header “X-API-KEY: mytoken12345678”

Reference: https://www.baeldung.com/spring-boot-api-key-secret

Encrypt Password In Properties For Spring Boot

It’s bad practice to keep passwords in application.properties for spring boot applications. At least, you should try something to make it less obvious. Jasypt is a handy tool to achieve this. It’s not bullet proof but it can hide obvious passwords, and others will take some effort to decrypt them.

  1. include maven dependencies for jasypt-spring-boot-start

        com.github.ulisesbocchio
        jasypt-spring-boot-starter
        3.0.5

  1. annotate your application @EnableEncryptableProperties

  2. in application.properties, use encrypted passwords using ENC(…)

secret.property=ENC(nrmZtkF7T0kjG/VodDvBw93Ct8EgjCA+)
  1. encrypt/decrypt in code
	@Override
	public String encrypt(String secret, String text) {
		PooledPBEStringEncryptor encryptor = new PooledPBEStringEncryptor();
		SimpleStringPBEConfig config = new SimpleStringPBEConfig();
		config.setPassword(secret);
		config.setAlgorithm("PBEWithMD5AndDES");
		config.setKeyObtentionIterations("1000");
		config.setPoolSize("1");
		config.setProviderName("SunJCE");
		config.setSaltGeneratorClassName("org.jasypt.salt.RandomSaltGenerator");
		config.setIvGeneratorClassName("org.jasypt.iv.RandomIvGenerator");
		config.setStringOutputType("base64");
		encryptor.setConfig(config);
		return encryptor.encrypt(text);
	}

	@Override
	public String decrypt(String secret, String cipher) {
		PooledPBEStringEncryptor encryptor = new PooledPBEStringEncryptor();
		SimpleStringPBEConfig config = new SimpleStringPBEConfig();
		config.setPassword(secret);
		config.setAlgorithm("PBEWithMD5AndDES");
		config.setKeyObtentionIterations("1000");
		config.setPoolSize("1");
		config.setProviderName("SunJCE");
		config.setSaltGeneratorClassName("org.jasypt.salt.RandomSaltGenerator");
		config.setIvGeneratorClassName("org.jasypt.iv.RandomIvGenerator");
		config.setStringOutputType("base64");
		encryptor.setConfig(config);
		return encryptor.decrypt(cipher);
	}
  1. configure default encryptor
jasypt.encryptor.password=hello
jasypt.encryptor.algorithm=PBEWithMD5AndDES
#jasypt.encryptor.key-obtention-iterations=1000
#jasypt.encryptor.pool-size=1
#jasypt.encryptor.provider-name=SunJCE
#jasypt.encryptor.salt-generator-classname=org.jasypt.salt.RandomSaltGenerator
#jasypt.encryptor.iv-generator-classname=org.jasypt.iv.RandomIvGenerator
#jasypt.encryptor.string-output-type=base64
  1. produce encrypted keys in step 3 Using password="abc123", encrypt a message "hellojasypt", and then decrypt it to validate.
java -cp C:\Users\wofon\.m2\repository\org\jasypt\jasypt\1.9.3\jasypt-1.9.3.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI input="hellojasypt" password="abc123" algorithm=PBEWithMD5AndDES
java -cp C:\Users\wofon\.m2\repository\org\jasypt\jasypt\1.9.3\jasypt-1.9.3.jar org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="QIMBpiPCmCiqrg8E2w+VGTHunWWu6gO2" password="abc123" algorithm=PBEWithMD5AndDES

Reference:
https://github.com/ulisesbocchio/jasypt-spring-boot
https://www.geeksforgeeks.org/how-to-encrypt-passwords-in-a-spring-boot-project-using-jasypt/
https://infobrisk.com/tech-insights/how-to-use-jasypt-in-spring-boot-application/

Source 60m Bar Via Java

I’ve decided to source 60m bars for futures contracts via Sina Finance for building golden source of market data, as described here.

The source provides data in Json, which can be easily converted into Objects via jacksonxml library, with a few tricks.

Trick 1. regexp
The response contains overheads mixed with JSON, so we need to strip it off in order to obtain the clean and tidy JSON. The following code sniplet takes Json content for OHLCVP.

	@Test
	public void test_read(){
		CharSource source = Files.asCharSource(new File(FILESTORAGE + File.separator + HISTORY), Charsets.US_ASCII);
		try {
			//encoding conversion is just taking too long to try. let's focus on SecurityHistory conversion
			List<String> lines = source.readLines();
			logger.error("count: {}", lines.size());
			for(String line : lines) {
				logger.error("line={}", line.length());
			}

			Pattern p = Pattern.compile("(.*)(\\Q=(\\E)(.*)\\Q);\\E");
			Matcher m = p.matcher(lines.get(1));
			if(m.find()){
				int count = m.groupCount();
				logger.error("count:{}", count);
				for(int i=0;i<=count;i++) {
					logger.error("token:{}", m.group(i));
				}
			}

2. jackson
ObjectMapper can parse the JSON string to objects conveniently, via the following function call:
mapper.readValue(json, new TypeReference<List<SinaFutureXminBarJson>>(){});
However, we need to define SinaFutureXminBarJson model class to hold data from json text, which should define a deserializer, JsonDateTimeDeserializer, which helps to convert string, such as “2020-10-15 22:00:00” to Joda object of LocalDateTime. 

			String json = m.group(3);
			ObjectMapper mapper = new ObjectMapper();
			mapper.configure(JsonParser.Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
			mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
			mapper.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));//2017-07-10 00:00:00

			try {
				List<SinaFutureXminBarJson> items = mapper.readValue(json, new TypeReference<List<SinaFutureXminBarJson>>(){});
				List<SinaFutureXminBar> retval = Lists.newArrayList();
				for(SinaFutureXminBarJson bar : items) {
					retval.add(new SinaFutureXminBar.Builder().build(bar));	
				}
				for(SinaFutureXminBar bar : retval) {
					logger.error("bar: {}", bar);
				}
			} catch (JsonParseException e) {
public class SinaFutureXminBarJson {
	@JsonDeserialize(using = JsonDateTimeDeserializer.class)
	private LocalDateTime d;
	private Double o;
	private Double h;
	private Double l;
	private Double c;
	private Long v;
	private Long p;

	public LocalDateTime getD() {
		return d;
	}
	public void setD(LocalDateTime d) {
		this.d = d;
	}
	public Double getO() {
		return o;
	}
	public void setO(Double o) {
		this.o = o;
	}
	public Double getH() {
		return h;
	}
	public void setH(Double h) {
		this.h = h;
	}
	public Double getL() {
		return l;
	}
	public void setL(Double l) {
		this.l = l;
	}
	public Double getC() {
		return c;
	}
	public void setC(Double c) {
		this.c = c;
	}
	
	public Long getV() {
		return v;
	}
	public void setV(Long v) {
		this.v = v;
	}
	public Long getP() {
		return p;
	}
	public void setP(Long p) {
		this.p = p;
	}
	public String toString() {
		return MoreObjects.toStringHelper(this.getClass())
			.add("d", d)
			.add("o", o)
			.add("h", h)
			.add("l", l)
			.add("c", c)
			.add("v", v)
			.add("p", p)
			.toString();
	}
}
public class JsonDateTimeDeserializer extends StdDeserializer<LocalDateTime>{
    private static final long serialVersionUID = 1L;
    private static DateTimeFormatter format = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss");

    public JsonDateTimeDeserializer() {
        this(null);
    }

    public JsonDateTimeDeserializer(Class<LocalDateTime> t) {
        super(t);
    }

    @Override    
    public LocalDateTime deserialize(JsonParser parser, DeserializationContext context)
            throws IOException, JsonProcessingException {

        String date = parser.getText();

        return format.parseDateTime(date).toLocalDateTime();
    }
}