스타벅스 아이스 아메리카노를 선물로 보내드리겠습니다.
virtualenv venv
pip install Flask psycopg2 SQLAlchemy pandas gunicorn
# -*- coding: utf-8 -*-
from flask import Flask
app = Flask(__name__)
@app.route("/")
def index():
return "Hello World!"
3개의 파일이 필요하다.
스타벅스 아이스 아메리카노를 선물로 보내드리겠습니다.!apt update -q
!apt-get install -q openjdk-11-jdk-headless
%env JAVA_HOME "/usr/lib/jvm/java-11-openjdk-amd64"
Hit:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease
Get:2 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease [3,626 B]
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Hit:4 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease
Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Ign:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease
Hit:8 http://ppa.launchpad.net/cran/libgit2/ubuntu bionic InRelease
Hit:9 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release
Hit:10 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic InRelease
Get:11 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Hit:12 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease
Get:14 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2,867 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1,075 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2,297 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3,302 kB]
Fetched 9,797 kB in 6s (1,534 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
49 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists...
Building dependency tree...
Reading state information...
openjdk-11-jdk-headless is already the newest version (11.0.15+10-0ubuntu0.18.04.1).
The following package was automatically installed and is no longer required:
libnvidia-common-460
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.
env: JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"
!pip install sklearn2pmml
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting sklearn2pmml
Downloading sklearn2pmml-0.84.2.tar.gz (6.3 MB)
[K |████████████████████████████████| 6.3 MB 5.1 MB/s
[?25hRequirement already satisfied: joblib>=0.13.0 in /usr/local/lib/python3.7/dist-packages (from sklearn2pmml) (1.1.0)
Requirement already satisfied: scikit-learn>=0.18.0 in /usr/local/lib/python3.7/dist-packages (from sklearn2pmml) (1.0.2)
Requirement already satisfied: sklearn-pandas>=0.0.10 in /usr/local/lib/python3.7/dist-packages (from sklearn2pmml) (1.8.0)
Requirement already satisfied: numpy>=1.14.6 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.18.0->sklearn2pmml) (1.21.6)
Requirement already satisfied: scipy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.18.0->sklearn2pmml) (1.4.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.18.0->sklearn2pmml) (3.1.0)
Requirement already satisfied: pandas>=0.11.0 in /usr/local/lib/python3.7/dist-packages (from sklearn-pandas>=0.0.10->sklearn2pmml) (1.3.5)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.11.0->sklearn-pandas>=0.0.10->sklearn2pmml) (2022.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.11.0->sklearn-pandas>=0.0.10->sklearn2pmml) (2.8.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.7.3->pandas>=0.11.0->sklearn-pandas>=0.0.10->sklearn2pmml) (1.15.0)
Building wheels for collected packages: sklearn2pmml
Building wheel for sklearn2pmml (setup.py) ... [?25l[?25hdone
Created wheel for sklearn2pmml: filename=sklearn2pmml-0.84.2-py3-none-any.whl size=6298569 sha256=f6a564303dd11e9ce38b6c7b4ec4e8a4632499c61d3ee69f40070a1f1983ef80
Stored in directory: /root/.cache/pip/wheels/bb/e4/71/d3c8f75fae8d7f387f82099ec8cdd6b83cf1dccaeb3561c7b6
Successfully built sklearn2pmml
Installing collected packages: sklearn2pmml
Successfully installed sklearn2pmml-0.84.2
from sklearn.datasets import load_diabetes
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn2pmml import PMMLPipeline, sklearn2pmml
import pandas as pd
# 데이터 불러오기
df = load_diabetes()
X = pd.DataFrame(columns = df.feature_names, data = df.get('data'))
y = pd.DataFrame(columns = ['target'], data = df.get('target'))
# 데이터셋 분리하기
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1, random_state=42)
# Pipeline 구축하기
pipeline = PMMLPipeline([ ('regressor', DecisionTreeRegressor()) ])
# 모형 학습 시키기
pipeline.fit(X_train, y_train)
PMMLPipeline(steps=[('regressor', DecisionTreeRegressor())])
from sklearn.metrics import mean_absolute_error
y_pred = pipeline.predict(X_test)
print('MAE: ', mean_absolute_error(y_pred, y_test))
MAE: 65.42222222222222
# 모델 내보내기
sklearn2pmml(pipeline, 'model.pmml', with_repr = True)
import org.pmml4s.model.Model;
import java.util.*;
public class Main {
private final Model model = Model.fromFile(Main.class.getClassLoader().getResource("model.pmml").getFile());
public Double getRegressionValue(Map<String, Double> values) {
Object[] valuesMap = Arrays.stream(model.inputNames())
.map(values::get)
.toArray();
Object[] result = model.predict(valuesMap);
return (Double) result[0];
}
public static void main(String[] args) {
Main main = new Main();
Map<String, Double> values = Map.of(
"age", 20d,
"sex", 1d,
"bmi", -100d,
"bp", -200d,
"s1", 1d,
"s2", 2d,
"s3", 3d,
"s4", 4d,
"s5", 5d,
"s6", 6d
);
double predicted = main.getRegressionValue(values);
System.out.println(predicted);
}
}
CREATE TABLE iris (
sepal_length INTEGER,
sepal_width INTEGER,
petal_length INTEGER,
petal_width INTEGER,
species varchar2(20)
);
LOAD DATA
INFILE 'iris.csv'
INTO TABLE iris
fields terminated by ','
(
sepal_length,
sepal_width,
petal_length,
petal_width,
species
)
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import sklearn
import seaborn as sns
import mlflow
import mlflow.sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, KFold
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, plot_roc_curve, confusion_matrix
print(f"numpy version {np.__version__}")
print(f"pandas version {pd.__version__}")
print(f"matplotlib version {mpl.__version__}")
print(f"seaborn version {sns.__version__}")
print(f"sklearn version {sklearn.__version__}")
print(f"MLFlow version {mlflow.__version__}")
numpy version 1.23.1
pandas version 1.4.3
matplotlib version 3.5.2
seaborn version 0.11.2
sklearn version 1.1.1
MLFlow version 1.27.0
DATA_PATH = "C:\\Users\\your_id\\Desktop\\mlops_tutorial\\data\\creditcard.csv"
df = pd.read_csv(DATA_PATH)
print(df.head())
Time V1 V2 V3 V4 V5 V6 V7 \
0 0.0 -1.359807 -0.072781 2.536347 1.378155 -0.338321 0.462388 0.239599
1 0.0 1.191857 0.266151 0.166480 0.448154 0.060018 -0.082361 -0.078803
2 1.0 -1.358354 -1.340163 1.773209 0.379780 -0.503198 1.800499 0.791461
3 1.0 -0.966272 -0.185226 1.792993 -0.863291 -0.010309 1.247203 0.237609
4 2.0 -1.158233 0.877737 1.548718 0.403034 -0.407193 0.095921 0.592941
V8 V9 ... V21 V22 V23 V24 V25 \
0 0.098698 0.363787 ... -0.018307 0.277838 -0.110474 0.066928 0.128539
1 0.085102 -0.255425 ... -0.225775 -0.638672 0.101288 -0.339846 0.167170
2 0.247676 -1.514654 ... 0.247998 0.771679 0.909412 -0.689281 -0.327642
3 0.377436 -1.387024 ... -0.108300 0.005274 -0.190321 -1.175575 0.647376
4 -0.270533 0.817739 ... -0.009431 0.798278 -0.137458 0.141267 -0.206010
V26 V27 V28 Amount Class
0 -0.189115 0.133558 -0.021053 149.62 0
1 0.125895 -0.008983 0.014724 2.69 0
2 -0.139097 -0.055353 -0.059752 378.66 0
3 -0.221929 0.062723 0.061458 123.50 0
4 0.502292 0.219422 0.215153 69.99 0
[5 rows x 31 columns]
normal = df[df['Class'] == 0].sample(frac=0.5, random_state=42).reset_index(drop=True)
anomaly = df[df['Class']==1]
normal_train, normal_test = train_test_split(normal, test_size = 0.2, random_state=42)
anomary_train, anomary_test = train_test_split(anomaly, test_size = 0.2, random_state=42)
normal_train, normal_validate = train_test_split(normal_train, test_size = 0.25, random_state=42)
anomary_train, anomary_validate = train_test_split(anomary_train, test_size = 0.25, random_state=42)
normal_train.shape, normal_validate.shape, anomary_train.shape, anomary_validate.shape
((85294, 31), (28432, 31), (294, 31), (99, 31))
X_train = pd.concat((normal_train, anomary_train))
X_test = pd.concat((normal_test, anomary_test))
X_validate = pd.concat((normal_validate, anomary_validate))
y_train = np.array(X_train["Class"])
y_test = np.array(X_test["Class"])
y_validate = np.array(X_validate["Class"])
X_train = X_train.drop("Class", axis = 1)
X_test = X_test.drop("Class", axis = 1)
X_validate = X_validate.drop("Class", axis = 1)
X_train.shape, X_validate.shape, X_test.shape, y_train.shape, y_validate.shape, y_test.shape
((85588, 30), (28531, 30), (28531, 30), (85588,), (28531,), (28531,))
scaler = StandardScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
X_validate = scaler.transform(X_validate)
mlflow.log_metric() 함수를 통해 지표를 로깅할 수 있음mlflow.log_artifact() 함수를 통해 그래프를 저장할 수 잇음.def train(sk_model, X_train, y_train):
sk_model = sk_model.fit(X_train, y_train)
train_acc = sk_model.score(X_train, y_train)
mlflow.log_metric("train_acc", train_acc)
print(f"Train Accuracy: (train_acc:.3%)")
def evaluate(sk_model, X_test, y_test):
eval_acc = sk_model.score(X_test, y_test)
preds = sk_model.predict(X_test)
auc_score = roc_auc_score(y_test, preds)
mlflow.log_metric("eval_acc", eval_acc)
mlflow.log_metric("auc_score", auc_score)
print(f"Auc Score : {auc_score:.3%}")
print(f"Eval Score : {eval_acc:.3%}")
roc_plot = plot_roc_curve(sk_model, X_test, y_test, name="Scikit-Learn ROC Curve")
plt.savefig("sklearn_roc_plot.png")
plt.show()
plt.clf()
conf_matrix = confusion_matrix(y_test, preds)
ax = sns.heatmap(conf_matrix, annot=True, fmt='g')
ax.invert_xaxis()
ax.invert_yaxis()
plt.ylabel("Actual")
plt.xlabel("Predicted")
plt.title("Confusion Matrix")
plt.savefig("sklearn_conf_matrix.png")
mlflow.log_artifact("sklearn_roc_plot.png")
mlflow.log_artifact("sklearn_conf_matrix.png")
# 모델 설정
sk_model = LogisticRegression(random_state=None, max_iter=400, solver='newton-cg')
# 실험 이름 설정
mlflow.set_experiment("sklearn_experiment")
# 해당 이름으로 실행 배치
with mlflow.start_run():
train(sk_model, X_train, y_train)
evaluate(sk_model, X_test, y_test)
# 하나의 MLFlow 실행 컨텍스트에서 모든 코드를 묶을 수 있음.
# 참조 : https://mlflow.org/docs/latest/models.html#model-customization
mlflow.sklearn.log_model(sk_model, 'log_reg_model')
# 본질적으로 모델과 지표가 로깅되는 현재 실행을 가져오고 출력함.
print("Model run: ", mlflow.active_run().info.run_uuid)
mlflow.end_run()
Train Accuracy: (train_acc:.3%)
Auc Score : 86.355%
Eval Score : 99.888%
C:\Users\your_id\Desktop\mlops_tutorial\venv\lib\site-packages\sklearn\utils\deprecation.py:87: FutureWarning: Function plot_roc_curve is deprecated; Function :func:`plot_roc_curve` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: :meth:`sklearn.metric.RocCurveDisplay.from_predictions` or :meth:`sklearn.metric.RocCurveDisplay.from_estimator`.
warnings.warn(msg, category=FutureWarning)
스타벅스 아이스 아메리카노를 선물로 보내드리겠습니다.import numpy as np
import pandas as pd
np.random.seed(0) # 시드값 고정
intercept = 4 # 절편
slope = 3 # 기울기
# 변동성 주기 위해 노이즈 생성
noise = np.random.randn(50, 1)
# 50개의 x값 생성
x = 5 * np.random.rand(50, 1) # 0과 5사이의 실숫값 50개 생성
y = slope * x + intercept + noise
# 데이터 프레임 생성
data = pd.DataFrame({'X': x[:, 0], 'Y': y[:, 0]})
print(data.head())
X Y
0 0.794848 8.148596
1 0.551876 6.055784
2 3.281648 14.823682
3 0.690915 8.313637
4 0.982912 8.816293
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.scatter(data['X'], data['Y'])
plt.show()
$ virtualenv venv
created virtual environment CPython3.9.1.final.0-64 in 475ms
creator CPython3Posix(dest=/Users/evan/Desktop/django-iris-tutorial/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/evan/Library/Application Support/virtualenv)
added seed packages: pip==22.1.1, setuptools==62.3.2, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
$ source venv/bin/activate
(venv) $
(venv) $ pip install jupyterlab sklearn django
(venv) $ python -m jupyterlab
# Module 불러오기
from pandas import read_csv
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
import pandas as pd
# 데이터셋 불러오기
df = pd.read_csv("data/iris.csv")
# 독립변수와 종속변수 분리
X = df[['sepal_length','sepal_width','petal_length','petal_width']]
y = df['classification']
# 훈련데이터와 종속데이터 분리
X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size=0.20, random_state=1)
# 모형 학습
model = SVC(gamma='auto')
model.fit(X_train, Y_train)
# 모형 예측
sepal_length = float(1.5)
sepal_width = float(5)
petal_length = float(4)
petal_width = float(3)
result = model.predict([[sepal_length,sepal_width,petal_length,petal_width]]) # input must be 2D array
print(result)
['Iris-virginica']
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/base.py:445: UserWarning: X does not have valid feature names, but SVC was fitted with feature names
warnings.warn(
pd.to_pickle(model, r'models/svc_model.pickle')
(venv) $ django-admin startproject iris
(venv) $ cd iris
(venv) $ python manage.py startapp predict
iris/settings.py를 열고 아래와 같이 수정한다..
.
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'predict',
]
.
.
iris/urls.py를 열고 아래와 같이 수정한다.from django.contrib import admin
from django.urls import path, include
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('predict.urls', namespace='predict'))
]
predict/urls.py를 새로 생성하고 아래와 같이 코드를 추가한다.from django.urls import path
from . import views
app_name = 'predict'
urlpatterns = [
path('', views.predict, name='predict'),
]
predict/views.py에서 predict 함수를 만들어 백엔드 처리를 진행하고, 최종 결괏값을 predict.html로 돌려주는 함수를 구현할 것이다.from django.shortcuts import render
# Create your views here.
def predict(request):
return render(request, 'predict.html', {})
predict/templates 폴더를 만들고, predict.html 파일을 새로 생성한다.hello
(venv) iris$ python manage.py runserver
C:\Users\your_name>mysql -uroot -p
Enter password: ****
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 19
Server version: 8.0.28 MySQL Community Server - GPL
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
homestead 로 지정했다.mysql> CREATE DATABASE homestead;
Query OK, 1 row affected (0.01 sec)
mysql> CREATE USER '{username}'@'localhost' IDENTIFIED BY '{password}';
mysql> CREATE USER '{username}'@'%' IDENTIFIED BY '{password}';
mysql> CREATE USER 'homestead'@'localhost' IDENTIFIED BY 'secret';
Query OK, 0 rows affected (0.02 sec)
mysql> GRANT ALL PRIVILEGES ON homestead.* TO 'homestead'@'localhost';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye
mysql -h127.0.0.1 -u{username} -p {database} # 예시
mysql -h127.0.0.1 -uhomestead -p homestead # 적용
C:\Users\your_name>mysql -h127.0.0.1 -uhomestead -p homestead
Enter password: ******
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 21
Server version: 8.0.28 MySQL Community Server - GPL
Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| homestead |
| information_schema |
+--------------------+
2 rows in set (0.00 sec)
import boto3
print(boto3.__version__)
1.23.5
bucket = boto3.resource('s3')
response = bucket.create_bucket(
Bucket = "your_bucket_name",
ACL="private", # public-read
CreateBucketConfiguration = {
'LocationConstraint' : 'ap-northeast-2'
}
)
print(response)
s3.Bucket(name='your_bucket_name')
client = boto3.client('s3')
response = client.create_bucket(
Bucket = "your_bucket_name",
ACL = "private",
CreateBucketConfiguration = {
'LocationConstraint' : 'ap-northeast-2'
}
)
print(response)
{'ResponseMetadata': {'RequestId': '1X0BAXRG653Q7Y61', 'HostId': 'WwKyxNBcd1V9x6D/WZn8twMKSWKBnkwVCPWtvarZvyNSSvqr7Q77J6OFAdWuYAwiv/nQfXoW/0U=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': 'WwKyxNBcd1V9x6D/WZn8twMKSWKBnkwVCPWtvarZvyNSSvqr7Q77J6OFAdWuYAwiv/nQfXoW/0U=', 'x-amz-request-id': '1X0BAXRG653Q7Y61', 'date': 'Wed, 25 May 2022 03:16:52 GMT', 'location': 'http://your_bucket_name.s3.amazonaws.com/', 'server': 'AmazonS3', 'content-length': '0'}, 'RetryAttempts': 0}, 'Location': 'http://your_bucket_name.s3.amazonaws.com/'}
