> For the complete documentation index, see [llms.txt](https://docs.graviti.cn/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.graviti.cn/dev-doc/tools/shi-li-yan-shi/shu-ju-wa-jue.md).

# 数据挖掘

基于TensorBay Action平台，本示例将搭建一个整合数据爬取、数据转化、数据提取和数据分析四步骤的工作流，帮助您快速了解格物钛数据平台

#### 1.新建数据集

a. 进入TensorBay，在个人或团队账户下，点击【新建数据集】 

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FjaowMz3MQqMYNVSi3PWj%2FCreateDataset.png?alt=media\&token=85f34828-6bd7-4833-91b6-1276c4934d08)

#### 2.配置密钥

a. 点击导航栏中的【开发者工具】，点击【新建AccessKey】并复制accesskey  

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2Ft4feKmKNj5XeFcgNL32z%2Faccesskey.png?alt=media\&token=4eb5bda1-7b0d-409d-9d4f-29a801f8a1e2)

b. 在新建数据集界面，点击【设置】->【自动化配置】->【新建密钥】  

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FX5jgj3Wuocjo6qIM5YOa%2Fconfig.png?alt=media\&token=0e0059d2-8d5d-4b70-af70-509269ad6445)

c. 新建密钥，密钥名为`accesskey`, 密钥值为a步骤复制的accesskey 

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2F1IhdGdUXWcJisE55hRmg%2Faccesskey2.png?alt=media\&token=e815b892-a02a-46a9-9287-0d3bb77f811b)

#### 3.创建工作流

a. 在新建数据集详情页，点击【自动化】->【新建工作流】  

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2Fn5I1YTZosalHOsdiijnX%2Fworkflow.png?alt=media\&token=75e00ecc-e718-4a43-9fe3-78eceb56e5f6)

b. 填写工作流名称（注： Workflow名称只能包含小写字母、数字以及减号，名称不得少于2位字符且首位不能为减号。）  

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FInOPsz6WlIS010nt3oYy%2Fworkflow_name.png?alt=media\&token=05c95b61-1ac1-4972-8a77-3ff26e374034)

c. 选择工作流触发机制【Trigger】（默认manual） 

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2F6Wz8uHO2JEYgsEazBOXg%2Fworkflow_trigger.png?alt=media\&token=fb3a5ffc-ee62-400b-9647-5e0d9b6546c3)

d. 设置工作流参数【Parameters】（注：本次示例参数来源于镜像设置的命令行参数，用以调整爬取论文的月份，默认为1。）

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FLeDrqXMgaG1vPImUEB3H%2Fworkflow_parameter.png?alt=media\&token=2e30f747-ee0f-49b3-80b0-766307d24abc)

e. 选择实例配置【Instance】  

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2F9oiPAS19h5I2Qr4ZGEoj%2Fworkflow_instance.png?alt=media\&token=38fd4efc-a30c-4443-99df-a8c2c16c060e)

f. 复制下列内容建立YAML文件

```
# 一个workflow由多个task构成，它们可以串行或并行运行.
tasks:
  # 本workflow包括四个task，分别为：scraper,pdf2txt,parser,statistics
  scraper:
    container:
         # 本task运行所依赖性的镜像名, 支持公开及私有的镜像仓库
      image: hub.graviti.cn/miner/scraper:2.3

      # 镜像运行后执行`./archive/run.py {{workflow.parameters.month}}`这条命令
      command: [python3]
      args: ["./archive/run.py", "{{workflow.parameters.month}}"]
  pdf2txt:
    # pdf2txt依赖于scraper，即仅在scraper运行结束后才开始运行
    dependencies:
      - scraper
    container:
      image: hub.graviti.cn/miner/pdf2txt:2.0
      command: [python3]
      args: ["pdf2txt.py"]
  parser:
    # parser依赖于pdf2txt，即仅在pdf2txt运行结束后才开始运行
    dependencies:
      - pdf2txt
    container:
      image: hub.graviti.cn/miner/parser:2.0
      command: [python3]
      args: ["parser.py"]
  statistics:
    # statistics依赖于parser, 即仅在parser运行结束后才开始运行
    dependencies:
      - parser
    container:
      image: hub.graviti.cn/miner/statistics:2.0
      command: [python3]
      args: ["statistics.py"]
```

 

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FM8jTaQk1P5DuhQ8Y3JE3%2Fworkflow_yaml.png?alt=media\&token=5d5b8f04-7eaa-4aeb-8ea6-97069f1f331e)

g. 点击【发布】后并点击【确定】 

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FD1RPKAhsNstk2ESDqIgA%2Fworkflow_public.png?alt=media\&token=9e5c95b3-cf4a-4a2b-bf48-d5c57ccbdaab)

#### 4.运行工作流

a. 点击【自动化】，回到工作流界面。选择刚创建好的工作流并点击【运行工作流】  

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FpobONVNo2wahcMmurWqA%2Fworkflow_run.png?alt=media\&token=a7a3ea06-4739-488a-a79c-0ea9200971f9)

b. 调整参数，比如将month对应参数调整成10（月份），并点击【运行】 

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2F0ZGurFkvqVToZVtmcFsM%2Fworkflow_run_para.png?alt=media\&token=64443b9b-d896-4986-a815-99084cd6556e)

#### 5.查看结果

a. 在工作流详情页，点击【用户日志】查看工作流运行日志 

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2F0yKd9a8iICP8zoLA4If4%2Fworkflow_log.png?alt=media\&token=5740bfeb-634d-44b4-b617-bc1ee603b326)

b. 在数据集详情页，点击【概要】->【数据预览】查看统计结果  

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2F45RKbLSRA3ilrpq6GMCd%2Foutcome1.png?alt=media\&token=256cc268-c4d0-438f-9a41-2b7ee19b2367)

![](https://1609727380-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MGHaHpCAkpKvpxq5ZJn%2Fuploads%2FYqaRW3GCQV8WPkDyLVPN%2Foutcome2.png?alt=media\&token=9b6a1f26-5138-4444-9419-2566984ee6a6)