首页 › MongoDB论坛 › ETL工具之kettle的使用(一) MongoDB->Mysql

卡点 發表於 2019-6-11 15:18:00

ETL工具之kettle的使用(一) MongoDB->Mysql

<h2>kettle简介：</h2>
<p>　　Kettle是一款国外开源的ETL工具，纯java编写，可以在Window、Linux、Unix上运行，数据抽取高效稳定。</p>
<p>　　Kettle 中文名称叫水壶，该项目的主程序员MATT 希望把各种数据放到一个壶里，然后以一种指定的格式流出。</p>
<p>　　Kettle这个ETL工具集，它允许你管理来自不同数据库的数据，通过提供一个图形化的用户环境来描述你想做什么，而不是你想怎么做。</p>
<p>　　Kettle中有两种脚本文件，transformation和job，transformation完成针对数据的基础转换，job则完成整个工作流的控制</p>
<h2>1：Kettle的下载与安装</h2>
<p>　　下载地址 ： https://community.hds.com/docs/DOC-1009855</p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">1) 下载下来是压缩版&nbsp; &nbsp;解压就可以使用&nbsp; 解压后双击 Spoon.bat&nbsp; 等几秒 即出现图形界面工具</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611141736480-1908414666.png"></p>
<h2>2：从MongoDB数据库的otherData文档 中 读取数据到 Mysql数据库t_other_data表 中</h2>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">1)在kettle中，mongodb input组件。</span></h3>
<p>　　<img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611143702158-1230206935.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">2)&nbsp;编辑组件</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611143841987-1429381260.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">3)&nbsp;输入连接信息</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611144111773-665231802.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">4)&nbsp;写查询语句</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611144451082-1203819825.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">点击预览 可以预览成功说明配置无误</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611144550021-1186544656.png"></p>
<h3>　<span style="color: rgba(255, 102, 0, 1)">　5) MongoDB-&gt;Mysql中MongoDB的配置就ok了&nbsp; &nbsp; 接下来配置表输出&nbsp; 即 mysql的配置</span></h3>
<p>　　双击 表输出</p>
<p>&nbsp;<img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611144845253-1141872221.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">6) 填写数据库连接信息</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611145124404-1332020442.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">7) 点击测试 看mysql配置的是否无误&nbsp;</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611145242644-1711530902.png"></p>
<p>　　看到successfully说明 mysql配置无误&nbsp; (这里最初我配置的时候报错&nbsp; 报错信息后面单拎出来记录)</p>
<h3>&nbsp;　　<span style="color: rgba(255, 102, 0, 1)">8) 配置Mysql</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611145946135-1312899479.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">9) 点击运行</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611150327084-1142820289.png"></p>
<h3>　　<span style="color: rgba(255, 102, 0, 1)">10) 查看Mysql中t_other_data表中数据</span></h3>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611150502213-1337647755.png"></p>
<p>&nbsp;</p>
<p><img src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611150427919-1968891603.png"></p>
<h2>&nbsp;3：第一次在配置好Mysql后&nbsp; 点击测试 Mysql是报错的&nbsp;</h2>
<div class="cnblogs_code"><img src="http://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif"><img id="code_img_opened_07b19cec-cd46-486a-bde9-279eebd8a991" class="code_img_opened lazyload" style="display: none" alt="" data-src="http://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif">
<div id="cnblogs_code_open_07b19cec-cd46-486a-bde9-279eebd8a991" class="cnblogs_code_hide">
<pre><span style="color: rgba(0, 0, 0, 1)">org.pentaho.di.core.exception.KettleDatabaseException:
Error occurred </span><span style="color: rgba(0, 0, 255, 1)">while</span><span style="color: rgba(0, 0, 0, 1)"> trying to connect to the database

Driver </span><span style="color: rgba(0, 0, 255, 1)">class</span> <span style="color: rgba(128, 0, 0, 1)">'</span><span style="color: rgba(128, 0, 0, 1)">org.gjt.mm.mysql.Driver</span><span style="color: rgba(128, 0, 0, 1)">'</span> could not be found, make sure the <span style="color: rgba(128, 0, 0, 1)">'</span><span style="color: rgba(128, 0, 0, 1)">MySQL</span><span style="color: rgba(128, 0, 0, 1)">'</span> driver (jar file) <span style="color: rgba(0, 0, 255, 1)">is</span><span style="color: rgba(0, 0, 0, 1)"> installed.
org.gjt.mm.mysql.Driver


    at org.pentaho.di.core.database.Database.normalConnect(Database.java:</span><span style="color: rgba(128, 0, 128, 1)">472</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.core.database.Database.connect(Database.java:</span><span style="color: rgba(128, 0, 128, 1)">370</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.core.database.Database.connect(Database.java:</span><span style="color: rgba(128, 0, 128, 1)">341</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.core.database.Database.connect(Database.java:</span><span style="color: rgba(128, 0, 128, 1)">331</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.createDatabaseNodes(XulDatabaseExplorerController.java:</span><span style="color: rgba(128, 0, 128, 1)">380</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerController.init(XulDatabaseExplorerController.java:</span><span style="color: rgba(128, 0, 128, 1)">130</span><span style="color: rgba(0, 0, 0, 1)">)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:</span><span style="color: rgba(128, 0, 128, 1)">62</span><span style="color: rgba(0, 0, 0, 1)">)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:</span><span style="color: rgba(128, 0, 128, 1)">43</span><span style="color: rgba(0, 0, 0, 1)">)
    at java.lang.reflect.Method.invoke(Method.java:</span><span style="color: rgba(128, 0, 128, 1)">498</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:</span><span style="color: rgba(128, 0, 128, 1)">313</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.initialize(AbstractXulDomContainer.java:</span><span style="color: rgba(128, 0, 128, 1)">141</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.swt.SwtXulRunner$</span><span style="color: rgba(128, 0, 128, 1)">1</span>.run(SwtXulRunner.java:<span style="color: rgba(128, 0, 128, 1)">67</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.eclipse.swt.widgets.Synchronizer.syncExec(Unknown Source)
    at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
    at org.pentaho.ui.xul.swt.SwtXulRunner.initialize(SwtXulRunner.java:</span><span style="color: rgba(128, 0, 128, 1)">64</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseExplorerDialog.open(XulDatabaseExplorerDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">92</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.core.database.dialog.DataOverrideHandler.explore(DataOverrideHandler.java:</span><span style="color: rgba(128, 0, 128, 1)">86</span><span style="color: rgba(0, 0, 0, 1)">)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:</span><span style="color: rgba(128, 0, 128, 1)">62</span><span style="color: rgba(0, 0, 0, 1)">)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:</span><span style="color: rgba(128, 0, 128, 1)">43</span><span style="color: rgba(0, 0, 0, 1)">)
    at java.lang.reflect.Method.invoke(Method.java:</span><span style="color: rgba(128, 0, 128, 1)">498</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:</span><span style="color: rgba(128, 0, 128, 1)">313</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:</span><span style="color: rgba(128, 0, 128, 1)">157</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:</span><span style="color: rgba(128, 0, 128, 1)">141</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.swt.tags.SwtButton.access$</span><span style="color: rgba(128, 0, 128, 1)">500</span>(SwtButton.java:<span style="color: rgba(128, 0, 128, 1)">43</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.ui.xul.swt.tags.SwtButton$</span><span style="color: rgba(128, 0, 128, 1)">4</span>.widgetSelected(SwtButton.java:<span style="color: rgba(128, 0, 128, 1)">137</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.eclipse.jface.window.Window.runEventLoop(Window.java:</span><span style="color: rgba(128, 0, 128, 1)">820</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.eclipse.jface.window.Window.open(Window.java:</span><span style="color: rgba(128, 0, 128, 1)">796</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">80</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">47</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">116</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">61</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.trans.step.BaseStepDialog.showDbDialogUnlessCancelledOrValid(BaseStepDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">808</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.trans.step.BaseStepDialog$EditConnectionListener.widgetSelected(BaseStepDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">1515</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.trans.steps.tableoutput.TableOutputDialog.open(TableOutputDialog.java:</span><span style="color: rgba(128, 0, 128, 1)">888</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:</span><span style="color: rgba(128, 0, 128, 1)">120</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:</span><span style="color: rgba(128, 0, 128, 1)">8662</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:</span><span style="color: rgba(128, 0, 128, 1)">3293</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:</span><span style="color: rgba(128, 0, 128, 1)">785</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:</span><span style="color: rgba(128, 0, 128, 1)">1381</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:</span><span style="color: rgba(128, 0, 128, 1)">7817</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:</span><span style="color: rgba(128, 0, 128, 1)">9179</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:</span><span style="color: rgba(128, 0, 128, 1)">707</span><span style="color: rgba(0, 0, 0, 1)">)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:</span><span style="color: rgba(128, 0, 128, 1)">62</span><span style="color: rgba(0, 0, 0, 1)">)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:</span><span style="color: rgba(128, 0, 128, 1)">43</span><span style="color: rgba(0, 0, 0, 1)">)
    at java.lang.reflect.Method.invoke(Method.java:</span><span style="color: rgba(128, 0, 128, 1)">498</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.commons.launcher.Launcher.main(Launcher.java:</span><span style="color: rgba(128, 0, 128, 1)">92</span><span style="color: rgba(0, 0, 0, 1)">)
Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Driver </span><span style="color: rgba(0, 0, 255, 1)">class</span> <span style="color: rgba(128, 0, 0, 1)">'</span><span style="color: rgba(128, 0, 0, 1)">org.gjt.mm.mysql.Driver</span><span style="color: rgba(128, 0, 0, 1)">'</span> could not be found, make sure the <span style="color: rgba(128, 0, 0, 1)">'</span><span style="color: rgba(128, 0, 0, 1)">MySQL</span><span style="color: rgba(128, 0, 0, 1)">'</span> driver (jar file) <span style="color: rgba(0, 0, 255, 1)">is</span><span style="color: rgba(0, 0, 0, 1)"> installed.
org.gjt.mm.mysql.Driver

    at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:</span><span style="color: rgba(128, 0, 128, 1)">515</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.core.database.Database.normalConnect(Database.java:</span><span style="color: rgba(128, 0, 128, 1)">456</span><span style="color: rgba(0, 0, 0, 1)">)
    ... </span><span style="color: rgba(128, 0, 128, 1)">66</span><span style="color: rgba(0, 0, 0, 1)"> more
Caused by: java.lang.ClassNotFoundException: org.gjt.mm.mysql.Driver
    at java.net.URLClassLoader.findClass(URLClassLoader.java:</span><span style="color: rgba(128, 0, 128, 1)">382</span><span style="color: rgba(0, 0, 0, 1)">)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:</span><span style="color: rgba(128, 0, 128, 1)">424</span><span style="color: rgba(0, 0, 0, 1)">)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:</span><span style="color: rgba(128, 0, 128, 1)">357</span><span style="color: rgba(0, 0, 0, 1)">)
    at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:</span><span style="color: rgba(128, 0, 128, 1)">490</span><span style="color: rgba(0, 0, 0, 1)">)
    ... </span><span style="color: rgba(128, 0, 128, 1)">67</span> more</pre>
</div>
<span class="cnblogs_code_collapse">View Code</span></div>
<p>错误信息显示是jar包未安装</p>
<p>下载mysql-connector-java-5.0.7 jar包并放到lib目录下 关闭图形界面重新打开Spoon.bat即可</p>
<p>(有个小坑 ： <span style="color: rgba(255, 102, 0, 1)">我第一次下载的版本是8依然报错，随后版本换成5好用了</span> )</p>
<p>下载jar包地址：&nbsp;http://mvnrepository.com/artifact/mysql/mysql-connector-java</p>
<p><img alt="" data-src="https://img2018.cnblogs.com/blog/800085/201906/800085-20190611151034923-260917917.png"></p>
<p>思考：MongoDB的数据读取到Mysql中是手动处理的&nbsp; 这在生产环境合理吗？如何自动实时的读取数据呢？下节总结</p>

</div>
<div id="MySignature" role="contentinfo">
    加油的人生不解释<br><br>
来源：https://www.cnblogs.com/s6-b/p/11003783.html

查看完整版本: ETL工具之kettle的使用(一) MongoDB->Mysql