Measuring the Quality of Data

Before you start to solve a problem (a numerical problem) you should check its conditioning. An ill-conditioned problem will produce very fragile and unreliable results – no matter how elegant and sophisticated solution you come up with it will be vulnerable. If decisions or strategies are based on fragile outcomes, those decisions and strategies will…

Complexity and Revisiting Anomaly Detection

Wikipedia: “In data mining, anomaly detection (also outlier detection) is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. Typically the anomalous items will translate to some kind of problem such as bank fraud, a structural defect, medical problems or errors in a text.…

Prediction: a quick way to look like an idiot.

What is a prediction? A prediction is a statement about what will happen or might happen in the future. This is the standard definition that you will find in a dictionary. Other, softer definitions, run like this: a statement about what you think will happen in the future. According to these two definitions – both are…

What is Critical Complexity?

Knowledge is an organized and dynamic set of interdependent rules. An example of a rule: “if UNEMPLOYMENT increases then NEW HOUSE CONSTRUCTION decreases”. This is an example of a fuzzy rule – no numbers just a global trend. Rules can be more or less fuzzy (or crisp) depending on how many experiments (data samples) they are…

Is It Possible To Make Predictions?

Prediction of the future has always been man’s dream. However, there is an overwhelming amount of physical evidence that this is quite impossible. This is because the future is permanently under construction. Therefore, as every second passes, the future is changed. The cause of this are the laws of physics. If the future were predictable…