hadoop-0.20.2 fs 命令指南

hadoop-0.20.2 fs 命令指南

hehaibolocal:hadoop-0.20.2-tmp hehaibo$ hadoop fs -help

hadoop fs is the command to execute fs commands. The full syntax is: 

hadoop fs [-fs <local | file system URI>] [-conf <configuration file>]

 [-D <property=value>] [-ls <path>] [-lsr <path>] [-du <path>]

 [-dus <path>] [-mv <src> <dst>] [-cp <src> <dst>] [-rm [-skipTrash] <src>]

 [-rmr [-skipTrash] <src>] [-put <localsrc> ... <dst>] [-copyFromLocal <localsrc> ... <dst>]

 [-moveFromLocal <localsrc> ... <dst>] [-get [-ignoreCrc] [-crc] <src> <localdst>

 [-getmerge <src> <localdst> [addnl]] [-cat <src>]

 [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>] [-moveToLocal <src> <localdst>]

 [-mkdir <path>] [-report] [-setrep [-R] [-w] <rep> <path/file>]

 [-touchz <path>] [-test -[ezd] <path>] [-stat [format] <path>]

 [-tail [-f] <path>] [-text <path>]

 [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]

 [-chown [-R] [OWNER][:[GROUP]] PATH...]

 [-chgrp [-R] GROUP PATH...]

 [-count[-q] <path>]

 [-help [cmd]]

-fs [local | <file system URI>]:  Specify the file system to use.

  If not specified, the current configuration is used, 

  taken from the following, in increasing precedence: 

   core-default.xml inside the hadoop jar file 

   core-site.xml in $HADOOP_CONF_DIR 

  'local' means use the local file system as your DFS. 

  <file system URI> specifies a particular file system to 

  contact. This argument is optional but if used must appear

  appear first on the command line.  Exactly one additional

  argument must be specified. 

-ls <path>:  List the contents that match the specified file pattern. If

  path is not specified, the contents of /user/<currentUser>

  will be listed. Directory entries are of the form 

   dirName (full path) <dir> 

  and file entries are of the form 

   fileName(full path) <r n> size 

  where n is the number of replicas specified for the file 

  and size is the size of the file, in bytes.

-lsr <path>:  Recursively list the contents that match the specified

  file pattern.  Behaves very similarly to hadoop fs -ls,

  except that the data is shown for all the entries in the

  subtree.

-du <path>:  Show the amount of space, in bytes, used by the files that 

  match the specified file pattern.  Equivalent to the unix

  command "du -sb <path>/*" in case of a directory, 

  and to "du -b <path>" in case of a file.

  The output is in the form 

   name(full path) size (in bytes)

-dus <path>:  Show the amount of space, in bytes, used by the files that 

  match the specified file pattern.  Equivalent to the unix

  command "du -sb"  The output is in the form 

   name(full path) size (in bytes)

-mv <src> <dst>:   Move files that match the specified file pattern <src>

  to a destination <dst>.  When moving multiple files, the 

  destination must be a directory. 

-cp <src> <dst>:   Copy files that match the file pattern <src> to a 

  destination.  When copying multiple files, the destination

  must be a directory. 

-rm [-skipTrash] <src>:  Delete all files that match the specified file pattern.

  Equivalent to the Unix command "rm <src>"

  -skipTrash option bypasses trash, if enabled, and immediately

deletes <src>

-rmr [-skipTrash] <src>:  Remove all directories which match the specified file 

  pattern. Equivalent to the Unix command "rm -rf <src>"

  -skipTrash option bypasses trash, if enabled, and immediately

deletes <src>

-put <localsrc> ... <dst>:  Copy files from the local file system 

  into fs. 

-copyFromLocal <localsrc> ... <dst>: Identical to the -put command.

-moveFromLocal <localsrc> ... <dst>: Same as -put, except that the source is

  deleted after it's copied.

-get [-ignoreCrc] [-crc] <src> <localdst>:  Copy files that match the file pattern <src> 

  to the local name.  <src> is kept.  When copying mutiple, 

  files, the destination must be a directory. 

-getmerge <src> <localdst>:  Get all the files in the directories that 

  match the source file pattern and merge and sort them to only

  one file on local fs. <src> is kept.

-cat <src>:  Fetch all files that match the file pattern <src> 

  and display their content on stdout.

-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>:  Identical to the -get command.

-moveToLocal <src> <localdst>:  Not implemented yet 

-mkdir <path>:  Create a directory in specified location. 

-setrep [-R] [-w] <rep> <path/file>:  Set the replication level of a file. 

  The -R flag requests a recursive change of replication level 

  for an entire tree.

-tail [-f] <file>:  Show the last 1KB of the file. 

  The -f option shows apended data as the file grows. 

-touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format

  in a file at <path>. An error is returned if the file exists with non-zero length

-test -[ezd] <path>: If file { exists, has zero length, is a directory

  then return 0, else return 1.

-text <src>:  Takes a source file and outputs the file in text format.

  The allowed formats are zip and TextRecordInputStream.

-stat [format] <path>: Print statistics about the file/directory at <path>

  in the specified format. Format accepts filesize in blocks (%b), filename (%n),

  block size (%o), replication (%r), modification date (%y, %Y)

-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...

  Changes permissions of a file.

  This works similar to shell's chmod with a few exceptions.

 -R modifies the files recursively. This is the only option

  currently supported.

 MODE Mode is same as mode used for chmod shell command.

  Only letters recognized are 'rwxX'. E.g. a+r,g-w,+rwx,o=r

 OCTALMODE Mode specifed in 3 digits. Unlike shell command,

  this requires all three digits.

  E.g. 754 is same as u=rwx,g=rx,o=r

  If none of 'augo' is specified, 'a' is assumed and unlike

  shell command, no umask is applied.

-chown [-R] [OWNER][:[GROUP]] PATH...

  Changes owner and group of a file.

  This is similar to shell's chown with a few exceptions.

 -R modifies the files recursively. This is the only option

  currently supported.

  If only owner or group is specified then only owner or

  group is modified.

  The owner and group names may only cosists of digits, alphabet,

  and any of '-_.@/' i.e. [-_.@/a-zA-Z0-9]. The names are case

  sensitive.

  WARNING: Avoid using '.' to separate user name and group though

  Linux allows it. If user names have dots in them and you are

  using local file system, you might see surprising results since

  shell command 'chown' is used for local files.

-chgrp [-R] GROUP PATH...

  This is equivalent to -chown ... :GROUP ...

-count[-q] <path>: Count the number of directories, files and bytes under the paths

  that match the specified file pattern.  The output columns are:

  DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME or

  QUOTA REMAINING_QUATA SPACE_QUOTA REMAINING_SPACE_QUOTA 

        DIR_COUNT FILE_COUNT CONTENT_SIZE FILE_NAME

-help [cmd]:  Displays help for given command or all commands if none

  is specified.

hehaibolocal:hadoop-0.20.2-tmp hehaibo$ 

Tomcat 进程意外退出-tomcat stopInternal

最近开发环境的tomcat总是莫名其妙的挂了。

Aug 04, 2018 10:50:04 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-18080"]
Aug 04, 2018 10:50:04 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-58080"]
Aug 04, 2018 10:50:04 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
20180804:10:50:04.547 [localhost-startStop-2] [org.springframework.context.support.AbstractApplicationContext-982] INFO Closing WebApplicationContext for namespace 'optimus-servlet': startup date [Sat Aug 04 10:07:59 CST 2018]; parent: Root WebApplicationContext
Aug 04, 2018 10:50:05 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
SEVERE: The web application [] registered the JDBC driver [com.alibaba.druid.proxy.DruidDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Aug 04, 2018 10:50:05 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesJdbc
SEVERE: The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Aug 04, 2018 10:50:05 AM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads

Aug 04, 2018 10:50:05 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-18080"]
Aug 04, 2018 10:50:05 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-58080"]
Aug 04, 2018 10:50:05 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-18080"]
Aug 04, 2018 10:50:05 AM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-58080"]

Google搜索引擎 找到答案:

转载自并发编程网 – ifeve.com  链接地址: Tomcat进程意外退出的问题分析

另外见:

tomcat server组件监听shutdown命令关闭服务器之源码分析

https://blog.csdn.net/joenqc/article/details/75212775

启动tomcat的脚本:

按照并发编程网大神的提示:加上

#使用监视模式
set -m

完美解决问题。

#!/usr/bin/env sh
# 参数1 分支 参数2 配置项
#使用监视模式
set -m
p1=18080;
p2=$1;
p3=$2;
if [ $# -lt 1 ]; then
 p2=develop
fi
if [ $# -lt 2 ]; then
 p3=xjf221
fi
sysname=test
cd `dirname $0`
cur_dir_temp=`pwd`;
cur_dir=$(dirname ${cur_dir_temp})
echo ${cur_dir}

git reset --hard
git clean -xdf
git checkout ${p2};
git pull;

cd ${cur_dir}

#配置Tomcat
cd `pwd`/script && sh setup_tomcat.sh ${sysname} ${p1};
cd ${cur_dir}

mvn clean install -DskipTests

ps auxwww | grep java | grep ${p1} | grep ${sysname} | awk '{print $2}' | xargs kill -9 2>/dev/null;
~/tomcat/tomcat_${sysname}_${p1}/bin/shutdown.sh ;
rm -rf ~/tomcat/tomcat_${sysname}_${p1}/webapps/ROOT.war;
rm -rf ~/tomcat/tomcat_${sysname}_${p1}/webapps/ROOT;
unzip ${cur_dir}/target/ROOT.war -d ~/tomcat/tomcat_${sysname}_${p1}/webapps/ROOT;
rm -rf catalina.pid 2>/dev/null;
~/tomcat/tomcat_${sysname}_${p1}/bin/startup.sh;

cd ~/tomcat/tomcat_${sysname}_${p1}/logs
tailf catalina.out

mysql创建用户和修改非Root用户密码

1 创建用户

create user 'test'@'%' identified by '123456';

flush privileges;

2 赋予用户数据库权限

grant all privileges on `testdb`.* to 'test'@'%' identified by '123456';
flush privileges;

3 修改非Root用户密码

https://dev.mysql.com/doc/refman/5.7/en/resetting-permissions.html

1 MySQL 5.7.6 and later:

ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';

2 MySQL 5.7.5 and earlier:

SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPass');

mac 下 hadoop-0.20.2单机版环境安装

1 安装java环境安装

略…

hehaibolocal:~ hehaibo$ java -version

java version "1.8.0_91"

Java(TM) SE Runtime Environment (build 1.8.0_91-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

hehaibolocal:~ hehaibo$ 

2 hadoop 安装

从 https://archive.apache.org/dist/hadoop/common/hadoop-0.20.2/hadoop-0.20.2.tar.gz 下载到 磁盘 /Users/hehaibo/hadoop/ 

执行

hehaibolocal:hadoop hehaibo$ tar xvf hadoop-0.20.2.tar.gz

安装后的目录如下:/Users/hehaibo/hadoop/hadoop-0.20.2

 

3 配置hadooop环境变量

sudo vi /etc/profile

输入:

HADOOP_HOME=/Users/hehaibo/hadoop/hadoop-0.20.2

PATH=".;$PATH:/usr/local/bin:$JAVA_HOME/bin:$ANT_HOME/bin:$MAVEN_HOME/bin:$HADOOP_HOME/bin

4 验证安装的版本

hehaibolocal:hadoop-0.20.2 hehaibo$ hadoop version

Hadoop 0.20.2

Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707

Compiled by chrisdo on Fri Feb 19 08:07:34 UTC 2010

hehaibolocal:hadoop-0.20.2 hehaibo$ 

5 配置hadoop环境

5.1 配置conf/core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<!-- 设置namenode所在主机,端口号是9000 -->

<name>fs.default.name</name>

<value>hdfs://localhost:9000/</value>

</property>

</configuration>

5.2 配置conf/hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<!-- 设置HDFS文件系统的元信息保存目录,可以设置多个,逗号分割 -->

<property>

<name>dfs.data.dir</name>

<value>/Users/hehaibo/hadoop/hadoop-0.20.2-tmp/hadoop-data</value>

</property>

<!-- 设置HDFS文件系统的数据保存在什么目录下,可以设置多个,逗号分割 -->

<property>

<name>dfs.name.dir</name>

<value>//Users/hehaibo/hadoop/hadoop-0.20.2-tmp/hadoop-name</value>

</property>

<property>

<!-- 设置数据块的复制次数,默认是3,如果slave节点数少于3,则写成相应的1或者2 -->

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

5.3 配置conf/mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<!-- 设置jobtracker所在机器,端口号9001 -->

<name>mapred.job.tracker</name>

<value>localhost:8021</value>

</property>

</configuration>

6 配置ssh免密码登录

% sudo apt-get install ssh

% ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa 

% cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

--免登成功
hehaibolocal:~ hehaibo$ ssh localhost

Last login: Thu Jul 19 16:30:48 2018

hehaibolocal:~ hehaibo$ 


7 修改conf/hadoop-env.sh 增加java环境变量

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/

8 hadoop 格式化hdfs文件系统

% hadoop namenode -format


hehaibolocal:~ hehaibo$ hadoop namenode -format

18/07/19 16:50:25 INFO namenode.NameNode: STARTUP_MSG: 

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = hehaibolocal.local/172.17.11.24

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 0.20.2

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010

************************************************************/

18/07/19 16:50:26 INFO namenode.FSNamesystem: fsOwner=hehaibo,staff,access_bpf,everyone,localaccounts,_appserverusr,admin,_appserveradm,_lpadmin,_appstore,_lpoperator,_developer,_analyticsusers,com.apple.access_ftp,com.apple.access_screensharing,com.apple.access_ssh-disabled

18/07/19 16:50:26 INFO namenode.FSNamesystem: supergroup=supergroup

18/07/19 16:50:26 INFO namenode.FSNamesystem: isPermissionEnabled=true

18/07/19 16:50:26 INFO common.Storage: Image file of size 97 saved in 0 seconds.

18/07/19 16:50:26 INFO common.Storage: Storage directory /Users/hehaibo/hadoop/hadoop-0.20.2-tmp/hadoop-name has been successfully formatted.

18/07/19 16:50:26 INFO namenode.NameNode: SHUTDOWN_MSG: 

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at hehaibolocal.local/172.17.11.24

************************************************************/

9 启动hadoop环境

9.1启动

hehaibolocal:~ hehaibo$ start-dfs.sh

namenode running as process 5375. Stop it first.

localhost: starting datanode, logging to /Users/hehaibo/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hehaibo-datanode-hehaibolocal.local.out

localhost: starting secondarynamenode, logging to /Users/hehaibo/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hehaibo-secondarynamenode-hehaibolocal.local.out

hehaibolocal:~ hehaibo$ start-mapred.sh 

starting jobtracker, logging to /Users/hehaibo/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hehaibo-jobtracker-hehaibolocal.local.out

localhost: starting tasktracker, logging to /Users/hehaibo/hadoop/hadoop-0.20.2/bin/../logs/hadoop-hehaibo-tasktracker-hehaibolocal.local.out

9.2查看启动进程

hehaibolocal:~ hehaibo$ jps

5603 DataNode

5669 SecondaryNameNode

5770 TaskTracker

5710 JobTracker

5375 NameNode

9.3 浏览器访问:

http://localhost:50070/dfshealth.jsp

http://localhost:50030/jobtracker.jsp

10 停止hadoop服务

hehaibolocal:~ hehaibo$ stop-dfs.sh 

stopping namenode

localhost: stopping datanode

localhost: stopping secondarynamenode

hehaibolocal:~ hehaibo$ stop-mapred.sh 

stopping jobtracker

localhost: stopping tasktracker

hehaibolocal:~ hehaibo$ 

 

Java开发牛人十大必备网站

以下是我收集的Java开发牛人必备的网站。这些网站可以提供信息,以及一些很棒的讲座, 还能解答一般问题、面试问题等。质量是衡量一个网站的关键因素,我个人认为这些网站质量都很好。接下来,我会跟大家分享我是如何使用这些网站学习和娱乐的。或许你会认为有些网站适合任何水平的开发者,但是我认为:对于Java开发牛人来说,网站的好坏取决于如何使用它们。

1、Stackoverflow

Stackoverflow.com 可能是编程界中最流行的网站了,该网站有成千上万个好问题和答案。学习API或者编程语言,通常依赖于代码示例,stackoverflow就可以提供大量的代码片段。

Stackoverflow的另一个优点在于它的社交性。你可以在一些特定的标签下查看问题,比如“Java”、“regex”,你就会看到哪些是高频问题。这对于学习,以及Java博主书写热门话题都是一种非常好的资源。

网站地址:http://stackoverflow.com/

2、DZone

这是一个非常有趣的网站,有相当多的开发者在这个网站上分享他们博客文章。就像一场冒险一样,在这里,你永远不知道下一秒会看到什么内容。

网站地址:http://www.dzone.com

3、LeetCode

如果有Java方面的面试问题,例如“在Java中,内存中的数组是什么样的”,你可以从很多Java教程中找到答案。但是,遇到类似于“如何将SortedArray转化成一个BalancedTree”这样的问题,你就应该去求助LeetCode了。LeetCode包含一系列算法问题, 它能为有关信息技术的面试提供一个社会性的平台。这个网站最好的地方就是,它可以通过大小不同的数据在线检查你的代码是否正确。很多人认为,面试成功的关键在于重复这三个重要步骤:编码->阅读->讨论。

网站地址: http://leetcode.com/

4、Java SE技术文档

这个网站包含了在使用Java SE API时会用到的所有文档,即使你是Java开发牛人,我敢肯定在这里你仍能发现一些有用的东西。比如,你可以阅读一些像“Java类的本质”、“部署”这样的教程。

网站地址: http://docs.oracle.com/javase/

5、Github

你也许知道可以免费的将你的项目托管在哪,但你不一定知道它还是一个学习当今流行的Java库和框架的优秀资源。比如,你想学 Spring MVC Framework,你可以搜索并且找到一些开源的项目。遵循“照葫芦画瓢”的原则,你能够通过实例快速的学习框架,尤其是你已经具有使用类似框架的经验。

网站地址:https://github.com/

6、Coursera

这是有关视频讲座的最好网站,在这里你可以找到许多顶尖学校的著名教授的优秀计算机科学课程。这些教授中,有些甚至是一些计算机科学领域的发明者。

网站地址:https://www.coursera.org/

7、Java World

这个网站包含了大量的Java教程,涉及各种各样的话题。其中很多文章都写的很好,并且有图片、插图进行说明。它可以被用来当作深度学习的教科书。

网站地址:http://www.javaworld.com/

8、IBM developerWorks

这里有很多由IBM公司的人写的非常不错的文章。

网站地址:http://www.ibm.com/developerworks/java/

9、Wikipedia 维基百科

这个网站是查询或学习几乎所有概念的最好的资源之一。例如,作为一个资深的Java开发人员,你可能只是想了解一些概念并不想深度研究。这里是一个可以免费查询最新信息的好地方。例如,什么是面向服务的编程?另外,它还能为你的写作提供大量素材。比如,同一个词在不同的领域可能会有完全不同的含义。有时,研究这些词在不同领域的不同含义是非常有意思的。

网站地址: http://en.wikipedia.org/wiki/

10、Program Creek

与以上几个网站相比,Programcreek.com在规模上要小很多。但是规模小也有优点,它在维护方面做的更好,读起来更有趣。在这里,你可以找到其他任何网站都不曾涉及到的话题,而且几乎每篇文章都包含漂亮的插图或者示例代码。它包含了来自不同领域的人发表的文章(科研、工业),并且经常为Java开发者提供优质材料。希望有一天,它也会出现在一些人的网站前十名排行名单中。

网站地址: http://www.programcreek.com/

注:当然,对于Java开发牛人来说还有很多其它不错的网站。如果你认为有其他的网站也应该列在这里,请留言,我以后也会把它们加进去。谢谢。

网友推荐:

感谢网友的热情推荐,这里会不断更新网友推荐的优秀网站。

原文链接: programcreek 翻译: ImportNew.com – 张文太
译文链接: http://www.importnew.com/7980.html
转载请保留原文出处、译者和译文链接。]