Summary of Researches on iData Lab

Published: by Creative Commons Licence

This is a summary of my researches during master life in iData Laboratory, School of Software Engineering, Tongji University.

Network Traffic Anomaly Detection

We applied several kinds of algorithms on detecting the anomalous IPs from the network traffic.

Propagation Based Detection

Here is the source code of this part. Basic idea is provided by one of my fellow students Shupei Wang1. I refined some of the iteration rules according to the shape of data.

Clustering

Another fellow Yudi Tang did some clusterings with our data [source code], while I applied affinity propagation clustering to the data after the preprocessing offered by Tang.

Email Anomaly Detection

We also find the email contact of Hacking Team. Here is the dataset. Our purpose is to find the email users who act anomalously.

We generated the graph below.

bassi

Predict Stock Market Indices

We predicted Dow Jones Industrial Average with the news selected from Reddit.

Our idea is described below.

for time in all_time:
	first[p] = RandomForestRegression(series[time-p, time])
	topic_distribution = dynamicLDA(documents[time])
	second = RandomForestRegression(topic_distribution)
for time in all_time:
	predict[p] = AutoRegressiveExogenous(first,second)

The result is shown below.

series

We submitted a paper in this topic2, and Our codes can be seen here. My graduate paper is also about this.

P.S. I have created my personal works to readthedocs in here.

  1. Shi Y., Wang S., Zhao Q., Li J. (2017) A Hybrid Approach of HTTP Anomaly Detection. In: Song S., Renz M., Moon YS. (eds) Web and Big Data. APWeb-WAIM 2017. Lecture Notes in Computer Science, vol 10612. Springer, Cham ↩

  2. Y. Zhong, Q. Zhao and W. Rao, "Predicting stock market indexes with world news," 2017 4th International Conference on Systems and Informatics (ICSAI), Hangzhou, China, 2017, pp. 1535-1540. doi: 10.1109/ICSAI.2017.8248528 ↩