sqlflow
SQLFlow is a bridge that connects a SQL engine, e.g. MySQL, Hive or MaxCompute, with TensorFlow, XGBoost and other machine learning toolkits. SQLFlow extends the SQL syntax to enable model training, prediction and model explanation.
The current experience of development ML based applications requires a team of data engineers, data scientists, business analysts as well as a proliferation of advanced languages and programming tools like Python, SQL, SAS, SASS, Julia, R. The fragmentation of tooling and development environment brings additional difficulties in engineering to model trainning/tunning. What if we marry the most widely used data management/processing language SQL with ML/system capabilities and let engineers with SQL skills develop advanced ML based applications?
There are already some work in progress in the industry. We can write
simple machine learning prediction (or scoring) algorithms in SQL using
operators like DOT_PRODUCT
.
However, this requires copy-n-pasting model parameters from the
training program to SQL statements. In the commercial world, we see some
proprietary SQL engines providing extensions to support machine
learning capabilities.
Microsoft SQL Server: Microsoft SQL Server has the machine learning service that runs machine learning programs in R or Python as an external script.
Teradata SQL for DL: Teradata also provides a RESTful service, which is callable from the extended SQL SELECT syntax.
Google BigQuery: Google BigQuery enables machine learning in SQL by introducing the CREATE MODEL
statement.
None of the existing solution solves our pain point, instead we want it to be fully extensible.
This solution should be compatible to many SQL engines, instead of a specific version or type.
It should support sophisticated machine learning models, including TensorFlow for deep learning and XGBoost for trees.
We also want the flexibility to configure and run cutting-edge ML algorithms including specifying feature crosses, at least, no Python or R code embedded in the SQL statements, and fully integrated with hyperparameter estimation.
Here are examples for training a Tensorflow DNNClassifer model using sample data Iris.train, and running prediction using the trained model. You can see how cool it is to write some elegant ML code using SQL:
sqlflow> SELECT *FROM iris.trainTO TRAIN DNNClassifier WITH model.n_classes = 3, model.hidden_units = [10, 20] COLUMN sepal_length, sepal_width, petal_length, petal_width LABEL class INTO sqlflow_models.my_dnn_model; ... Training set accuracy: 0.96721Done training
sqlflow> SELECT *FROM iris.testTO PREDICT iris.predict.class USING sqlflow_models.my_dnn_model; ... Done predicting. Predict table : iris.predict
SQLFlow will love to support as many mainstream ML frameworks and data sources as possible, but we feel like the expansion would be hard to be done merely on our own, so we would love to hear your options on what ML frameworks and data sources you are currently using and build upon. Please refer to our roadmap for specific timelines, also let us know your current scenarios and interests around SQLFlow project so we can prioritize based on the feedback from the community.
Your feedback is our motivation to move on. Please let us know your questions, concerns, and issues by filing Github Issues.
还没有评论,说两句吧!
热门资源
Keras-ResNeXt
Keras ResNeXt Implementation of ResNeXt models...
seetafaceJNI
项目介绍 基于中科院seetaface2进行封装的JAVA...
spark-corenlp
This package wraps Stanford CoreNLP annotators ...
capsnet-with-caps...
CapsNet with capsule-wise convolution Project ...
inferno-boilerplate
This is a very basic boilerplate example for pe...
智能在线
400-630-6780
聆听.建议反馈
E-mail: support@tusaishared.com