Machine Learning (ML) uses are so close to everyone now! Look at our inseparable best friend called ‘Smartphone’. This little device is filled with many softwares and apps using machine learning to secretly monitor our usage patterns. Optimising the phone’s performance and improving our experience with it.
Do a search online. You will find articles predicting the future jobs market. Where many will be replaced by intelligent systems capable of performing knowledge work and automation. Bright or gloomy? I am not sure…
Many companies have started data mining projects for predictive insights or set up machine learning departments to find hidden gems in their trove of accumulated data.
A topic on Machine learning is too big to cover in one post. Here you will get short answers to three questions.
1) Why is it so popular? 2) How to go about using machine learning? 3) What it means to us?
Why is it so popular?
So why the hype now when people have been using Machine Learning since 1950? Applications of it have mostly been within academical interest until recent decade. Before its rapid adoption by businesses, the financial industry is probably the pioneer to commercially develop the use of machine learning in quantitative trading and profiting from it. Other than creating algorithms that trade profitably, among others uses are risks assessment on loans default and investments profitability.
During the early days of machine learning, when floppy disk made in 1986 has a storage of 1.44MB, there wasn’t enough data to feed the system. It was unlikely to have many instances when a machine learning problem can learn a highly optimised algorithm. Even when there are data, hair wrenching computation speed back then will make any attempt to process these data an arduous process. Adding to the fact that after investing so much time and resources, you might not successfully create a good optimised algorithm. Not difficult to see why its uses are unpopular outside of the academic world.
Came the big data explosion fuelled by the internet, coupled with advancement on computers processing speed cleared the two big barriers. We have got better tools and the richer data amassed by the interconnected world, allowing us to mine valuable nuggets of insight. Interest to develop machine learning efficiency increases when insightful patterns and predictive trends are found in these data.
It turns out Machine Learning using statistical learning is an effective method to achieve predictive models with high accuracy. Machine learning system can learn a suitable algorithm for the task, without needing an expert to create a fix algorithm code. As condition changes, the algorithm continuously changes with the new information.
There is no need to learn to sort numbers. We already have algorithms for that, but there are many applications for which we do not have an algorithm but have lots of data.
For companies hoping to gain an extra edge through insights from their vast data collected, adopting machine learning become an important move.
How to go about using Machine Learning?
Machine learning is not the solution for all types of data analysis! Depending on the purpose of analysing your data, most data sorting and summarising tasks are better suited for common analytics tool (eg: histogram, waterfall and pie chart).
If you think artificial intelligence and machine learning are the same, their definitions are actually different. Artificial intelligent (AI) is the capability of a machine to imitate intelligent human behaviour and machine learning gives computers the ability to learn without being explicitly programmed. Use any ways to create a system that imitates intelligent human behaviour and that system has artificial intelligence. A product or service using artificial intelligent may have one or more machine learning modules in its operating system.
For those who are keen to explore learning Machine Learning, I have recently done a popular course from Coursera and wrote a short post on what I learnt here.
Broadly you will be working with math, data analysis, programming and a good amount of computing power.
My layman explanation:
Maths commonly needed in machine learning problem are Linear Algebra, Probability Theory & Statistics calculus and Calculus. Working with calculus always remind me of drawing lines on a graph paper when studying elementary math. Don’t be misled, for complex or innovative tasks, higher level math is commonly needed.
Data analysis is a lot about selecting, sorting, reviewing and managing data and its structure. This is an important part to make sure you’re using the most relevant data for optimal learning – on both algorithm accuracy and computational time.
Programming is to engineer how the system will learn. From processing dataset, monitoring and troubleshooting during training, to visualising optimisation results.
Computing time for your algorithm to learn meaningfully can vary greatly with two main factors. The size of data which can sometimes get astronomical. Some learning methods may need more computing time than others when optimising algorithms.
Size of data. Understanding social behaviours and genomics are areas where a huge amount of data in millions up to billions sets are often needed to achieve high accuracy.
Learning methods. Computing time varies with the method used, complex problems using deep learning often need more computing time.
Speech and face recognitions, self-driving car and DNA sequencing. These are some tasks that require both massive data and many hours of computer learning to reach reliable accuracy.
As a beginner looking at the math topics, programming and complexity it can get, I felt incapable after finishing my machine learning course. This very consoling blog from Sharp Sight Labs shine some lights to me on the prerequisite of machine learning. I feel less crippled after reading. Sharing an infographic I like on the blog.
Machine learning is very suitable (in my simple view) to automate tasks that:
- Involve predicting outcome basing on a list of attributes – eg: Predicting property price, selling price for new products.
- Make decisions by using probability to choose from a specific range of outcomes – eg: Speech and image recognition, classifying email categories, stocks trading.
- Find consistent or inconsistent patterns from a situation that is repetitive and possibly have some degrees of variations – eg: Anomaly detection used in cyber security and monitoring manufacturing line performance, categorising news articles, customer behaviour on buying.
- Mix above methods together to create some cool stuffs with artificial intelligent
- Of course those I have not thought of…
What does this mean to us?
In recent years, harvesting big data using machine learning have given rise to many systems with artificial narrow intelligence. Now, it is necessary to use machine learning in almost any systems harnessing artificial intelligence. With only more sophisticated artificial intelligence products or services to come.
For the future advancement of artificial intelligence, there are two polarised views from experts in this field. Optimists who believed we will eventually create a superintelligence in time and might even accidentally cause human extinction (this part is not encouraging). Pessimists who feel it will be difficult to even build a system with artificial general intelligence (human level intelligence) and understanding emotional intelligence. Hence, creating superintelligence is impossible. You can read this blog for leisure to get some good explanations and interesting possibilities on artificial intelligence.
I do agree with what Andrew Ng said during an interview.
People often ask me, “Andrew, what industries do you think AI will transform?”
I usually answer that it might be easier to think about what industries AI will not transform. To be honest, I struggled to think of one.
– Andrew Ng, Co-founder, Coursera; Adjunct Professor, Stanford University; formerly head of Baidu AI Group/Google Brain
Our use of devices to connect socially and speed up working during the information age are evolving. We are already expanding toward connecting different devices to enrich personal experiences, managing lifestyle/health and improving our quality of life. In this Internet of Things era, ever more devices will get connected and be communicating with one another, creating an invisible orchestration that immerses us with them. Making sense and managing the vast amount of data from end to end will rely greatly on machine learning.
I do think machine learning is going to affect everyone’s life in a big way and hopefully more for the better. Many jobs will probably be replaced. Perhaps these jobs are unlikely in any near future to be replaced and some more.