Linux常用命令11Linux文本处理示例

0.AWK常变量含义

常变量名 含义
ARGC 命令行变元个数
ARGV 命令行变元数组
FILENAME 当前输入文件名
FNR 当前文件中的记录号
FS 输入域分隔符,默认为一个空格
RS 输入记录分隔符
NF 当前记录里域个数
NR 到目前为止记录数
OFS 输出域分隔符
ORS 输出记录分隔符

1、统计文本中单词数量,并进行排序

grep -Eo "[a-z|A-Z]+" words.txt|awk '{word_cound[$1]++}END {for(aword in word_cound){print aword,word_cound[aword]|"sort -rn -k2"}}'

2、电话号码验证
规则:
A、(xxx) xxx-xxxx
B、xxx-xxx-xxxx

grep -Eo '^(\([0-9]{3}\) |[0-9]{3}-)[0-9]{3}-[0-9]{4}$' phonelist.txt

3、行列转换

awk '{for(i=0;++i<=NF;)t[i]=t[i]?t[i] FS $i:$i}END {for(i=0;i++<NF;)print t[i]}'  transport.txt

4、输出第10行

awk 'NR==10{print}' tenth.txt

ibatis调用了错误的mapping

昨天,公司一位同事遇到了十分神奇的一件事情,就是ibaits在应该调用A类A1方法时,ibatis会调用到B类的B1方法。

几个人查了半天,没发现情况。

最后细心的小江同学发现,是有一个接口文件的声明,与映射文件中的参数类型不一致造成的。

好坑啊。。。

ASP.Net检测到有潜在危险的Request.Form值

前两天公司一哥们遇到了这个问题,记录一下解决方法:

ASP.NET 2.0
方案一:将aspx文件中的page项添加ValidateRequest=”false”

  <%@ Page ValidateRequest="false"  Language="C#" AutoEventWireup="true" CodeFile="xxx.aspx.cs" Inherits="xxx.xxx" %>  

方案二:修改web.config配置文件(全局有效,慎用)

  <system.web> 
      <pages validateRequest="false"></pages>   
  </system.web> 

ASP.NET 4.0
方案一:修改web.config配置文件(全局有效,慎用)

  <system.web>
      <httpRuntime requestValidationMode="2.0" />
      <pages validateRequest="false"></pages>
  </system.web>

ASP.NET MVC
方案一:修改web.config配置文件(全局有效,慎用)
修改web.config配置文件(全局有效,慎用)

  <system.web>
      <httpRuntime requestValidationMode="2.0" />
  </system.web>

然后修改控制器

[HttpPost]
[ValidateInput(false)]
public ActionResult XXX(xxx xxx)
{
}

JSP输出容器的JVM参数

<%@ page language="java" import="java.util.*, java.lang.management.*" pageEncoding="UTF-8"%>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>系统参数输出</title>
<style type="text/css">
table
{
    border-collapse:collapse;
}
td
{
    border:1px solid #427BD6;
}
</style>
</head>

<body style="background-color:#f2f2f2;">

<h1>环境变量</h1>
<table>
<%
    Map<String, String> map = System.getenv();
    //RuntimeMXBean runtimeMxBean = ManagementFactory.getRuntimeMXBean();
    //Map<String, String> map = runtimeMxBean.getSystemProperties();
    for(String key : map.keySet())
    {
        out.println("<tr>");
        out.println("<td>");
        out.println(key);
        out.println("</td>");
        out.println("<td>");
        out.println(map.get(key));
        out.println("</td>");
        out.println("</tr>");
    }     
%>
</table>

<h1>JVM参数</h1>
<table>
<% 
    RuntimeMXBean runtimeMxBean = ManagementFactory.getRuntimeMXBean();
    
    out.println("<tr>");
    out.println("<td>");
    out.println(runtimeMxBean.getVmVendor()+" || "+runtimeMxBean.getVmName() +" || "+ runtimeMxBean.getVmVersion());
    out.println("</td>");
    out.println("</tr>");
    
    out.println("<tr>");
    out.println("<td>");
    out.println(runtimeMxBean.getSpecVendor()+" || "+runtimeMxBean.getSpecName() +" || "+ runtimeMxBean.getSpecVersion());
    out.println("</td>");
    out.println("</tr>");
    
    out.println("<tr>");
    out.println("<td>");
    out.println(runtimeMxBean.getName()+" || "+runtimeMxBean.getStartTime() +" || "+ runtimeMxBean.getUptime());
    out.println("</td>");
    out.println("</tr>");
    
    out.println("<tr>");
    out.println("<td>");
    out.println("MaxMemory="+Runtime.getRuntime().maxMemory()/1024/1024 + "m ");
    out.println("TotalMemory="+Runtime.getRuntime().totalMemory()/1024/1024 + "m ");
    out.println("FreeMemory="+Runtime.getRuntime().freeMemory()/1024/1024 + "m ");
    out.println("</td>");
    out.println("</tr>");
    
    List<MemoryPoolMXBean> memoryPoolMXBeans = ManagementFactory.getMemoryPoolMXBeans();
    for (MemoryPoolMXBean bean : memoryPoolMXBeans)
    {
        out.println("<tr>");
        out.println("<td>");
        out.println(bean.getName()+" || "+bean.getType()+" || Init="+bean.getUsage().getInit()/1024/1024+
        "m Max="+bean.getUsage().getMax()/1024/1024+"m Used="+bean.getUsage().getUsed()/1024/1024+
        "m Committed="+bean.getUsage().getCommitted()/1024/1024+"m ");
        out.println("</td>");
        out.println("</tr>");
    }
    
    List<String> arguments = runtimeMxBean.getInputArguments();
    for(String arg:arguments)
    {
        out.println("<tr>");
        out.println("<td>");
        out.println(arg);
        out.println("</td>");
        out.println("</tr>");
    }
%>
</table>

<h1>系统参数</h1>
<table>
<%
    Properties props = System.getProperties();
    for(Object o:props.keySet())
    {
        out.println("<tr>");
        out.println("<td>");
        out.println(o);
        out.println("</td>");
        out.println("<td>");
        out.println(props.get(o));
        out.println("</td>");
        out.println("</tr>");
    }
%>
</table>
</body>
</html>

XCode7编译Osirix

1、到Github下载源码
https://github.com/pixmeo/osirix

2、用Xcode打开Osirix.xcodeproj,提示要升级配置,升级

3、运行任务Unzip Binaries

4、Osirix目标调整为x86(上面的Binaries都是x86的)

5、编译Osirix,恩,出错了是吧

6、下载openssl库

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install openssl

将/usr/local/opt/openssl/include添加到include路径
将/usr/local/opt/openssl/lib添加到lib路径

7、在Osirix项目中,去除Message依赖

8、还需要调整几个编译错误和几个连接错误,然后就好了
其中有一个错误是有多个jaritab符号,可以找到三个,然后改为不同名或改为static就好了
其余问题都是很简单的问题咯

9、搞定

Java四种引用方式

最近写了个例子,说明了一下强引用、软引用、弱引用、虚引用的区别。

1、NString.java

package com.neohope.reference;

/**
 * Created by Hansen
 */
public class NString {
    private String name;
    private String value;

    public NString(String name, String value)
    {
        this.name = name;
        this.value=value;
    }

    public NString(String name, StringBuilder builder)
    {
        this.name = name;
        this.value=builder.toString();
    }

    public String Name()
    {
        return name;
    }

    public String Value()
    {
        return value;
    }

    @Override
    protected void finalize()
    {
        System.out.println(">> "+Name()+" finalize called");
        try {
            super.finalize();
        } catch (Throwable throwable) {
            throwable.printStackTrace();
        }
    }
}

2、TestReferenceA.java

package com.neohope.reference;

import java.lang.ref.PhantomReference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.SoftReference;
import java.lang.ref.WeakReference;
import java.util.WeakHashMap;

/**
 * Created by Hansen
 * JVM参数:
 * -XX:+PrintGCDetails
 * -Xms6m -Xmx12m
 */
public class TestReferenceA {
    public static void main(String[] args) {
        //申请内存
        Runtime rt=Runtime.getRuntime();
        System.out.println("Total memory is "+rt.totalMemory());
        int MemSize = 1024*1024*1;
        StringBuilder builder = new StringBuilder(MemSize);
        for (int i=0;i<MemSize/8;i++) {
            builder.append("ABCDEFGH");
        }
        System.out.println("Total memory is "+rt.totalMemory());

        //用s00与s01两个大字符串演示强引用与软引用的区别
        NString s00 = new NString("s00",builder);
        NString s01 = new NString("s01",builder);
        //s02与s03演示WeakReference的两种用法
        NString s02 = new NString("s02","s03");
        String key03 = new String("k03");
        NString s03 = new NString("s03","s03");
        //s04演示PhantomReference
        NString s04 = new NString("s04","s04");
        System.out.println("Total memory is "+rt.totalMemory());

        //第1类、强引用的对象,也最常用到的类型,内存不足时JVM将抛出异常也不要释放这些内存
        NString strongRef = s00;
        //第2类、只有软引用的数据,内存不足时JVM会回收这些内存
        SoftReference<NString> softRef = new SoftReference<NString>(s01);
        //第3类、只有弱引用的数据,无论内存是否充足JVM会回收这些内存,也就是弱引用不会影响对象的生命周期
        WeakReference<NString> weakRef = new WeakReference<NString>(s02);
        //WeakHashMap很有用,当key022为null时,s022也就会被GC咯
        WeakHashMap weakHashMap = new WeakHashMap();
        weakHashMap.put(key03,s03);
        //第4类、只有虚引用的数据,一开始refQueue为空,被GC才会有值
        ReferenceQueue refQueue = new ReferenceQueue<String>();
        PhantomReference<NString> phantomRef = new PhantomReference<NString>(s04, refQueue);

        //有强引用时,输出初始化情况
        System.out.println("With strong ref strongRef is " + (strongRef==null?"null":"not null"));
        System.out.println("With strong ref softRef is " + (softRef.get()==null?"null":"not null"));
        System.out.println("With strong ref weakRef is " + (weakRef.get()==null?"null":"not null"));
        System.out.println("With strong ref weakHashMap is " + (weakHashMap.isEmpty() ?"empty":"not empty"));
        //这里为null,当对象被GC时,会得到虚引用对象
        System.out.println("With strong ref refQueue has " + refQueue.poll());
        System.out.println();

        //去掉初始的强引用
        s00 = null;
        s01 = null;
        s02 = null;
        s03 = null;
        s04 = null;

        //多数情况下,没有来得及GC
        System.out.println("Before gc strongRef is " + (strongRef==null?"null":"not null"));
        System.out.println("Before gc softRef is " + (softRef.get()==null?"null":"not null"));
        System.out.println("Before gc weakRef is " + (weakRef.get()==null?"null":"not null"));
        System.out.println("Before gc weakHashMap is " + (weakHashMap.isEmpty() ?"empty":"not empty"));
        //null,当对象被GC时,会得到虚引用对象
        System.out.println("Before gc refQueue has " + refQueue.poll());
        System.out.println();

        //强制GC
        System.gc();
        System.out.println("After first gc strongRef is " + (strongRef==null?"null":"not null"));
        System.out.println("After first gc softRef is " + (softRef.get()==null?"null":"not null"));
        //null,weakRef已经被释放了
        System.out.println("After first gc weakRef is " + (weakRef.get()==null?"null":"not null"));
        System.out.println("After first gc weakHashMap is " + (weakHashMap.isEmpty() ?"empty":"not empty"));
        //与运行环境及GC方法有关,会出现一次有值的情况,然后一直为null
        System.out.println("After first gc refQueue has " + refQueue.poll());
        System.out.println();

        //将key03设为null,强制GC
        key03 = null;
        System.gc();
        System.out.println("After second gc strongRef is " + (strongRef==null?"null":"not null"));
        System.out.println("After second gc softRef is " + (softRef.get()==null?"null":"not null"));
        //null
        System.out.println("After second gc weakRef is " + (weakRef.get()==null?"null":"not null"));
        //empty,由于key被设为空,则会释放掉弱引用对象了
        System.out.println("After second gc weakHashMap is " + (weakHashMap.isEmpty() ?"empty":"not empty"));
        //与运行环境及GC方法有关,会出现一次有值的情况,然后一直为null
        System.out.println("After second gc refQueue has " + refQueue.poll());
        System.out.println();

        //Finalization
        System.runFinalization();
        System.out.println("After Finalization strongRef is " + (strongRef==null?"null":"not null"));
        System.out.println("After Finalization softRef is " + (softRef.get()==null?"null":"not null"));
        //null
        System.out.println("After Finalization weakRef is " + (weakRef.get()==null?"null":"not null"));
        //empty
        System.out.println("After Finalization weakHashMap is " + (weakHashMap.isEmpty() ?"empty":"not empty"));
        //与运行环境及GC方法有关,会出现一次有值的情况,然后一直为null
        System.out.println("After Finalization refQueue has " + refQueue.poll());
        System.out.println();

        //申请内存,造成内存不足
        System.out.println("Total memory is "+rt.totalMemory());
        StringBuilder builder2 = new StringBuilder(MemSize);
        for (int i=0;i<MemSize/8;i++) {
            builder2.append("ABCDEFGH");
        }
        NString tmp0 = new NString("tmp0", builder2);
        System.out.println("Total memory is "+rt.totalMemory());

        System.out.println("When mem is low strongRef is " + (strongRef==null?"null":"not null"));
        //null,内存不足时软引用会被释放
        System.out.println("When mem is low softRef is " + (softRef.get()==null?"null":"not null"));
        //null
        System.out.println("When mem is low weakRef is " + (weakRef.get()==null?"null":"not null"));
        //empty
        System.out.println("When mem is low weakHashMap is " + (weakHashMap.isEmpty() ?"empty":"not empty"));
        //与运行环境及GC方法有关,会出现一次有值的情况,然后一直为null
        System.out.println("When mem is low refQueue has " + refQueue.poll());
        System.out.println();

        //再次申请内存,JMV会内存爆掉
        NString tmp1 = null;
        try
        {
            tmp1 = new NString("tmp1",builder2);
        }
        catch(OutOfMemoryError ex)
        {
            System.out.println("Not enough memory");
        }

        //null
        System.out.println("tmp1 is " + (tmp1==null?"null":"not null"));
        //就算是抛出OutOfMemoryError,仍然不会释放
        System.out.println("When not enough memory strongRef is " + (strongRef==null?"null":"not null"));
        //null
        System.out.println("When not enough memory softRef is " + (softRef.get()==null?"null":"not null"));
        //null
        System.out.println("When not enough memory weakRef is " + (weakRef.get()==null?"null":"not null"));
        //empty
        System.out.println("When not enough memory weakHashMap is " + (weakHashMap.isEmpty() ?"empty":"not empty"));
        //null
        System.out.println("When not enough memory refQueue has " + refQueue.poll());
        System.out.println();
    }
}

3、输出

Total memory is 6094848
Total memory is 6094848
Total memory is 11390976
With strong ref strongRef is not null
With strong ref softRef is not null
With strong ref weakRef is not null
With strong ref weakHashMap is not empty
With strong ref refQueue has null

Before gc strongRef is not null
Before gc softRef is not null
Before gc weakRef is not null
Before gc weakHashMap is not empty
Before gc refQueue has null

After first gc strongRef is not null
After first gc softRef is not null
After first gc weakRef is null
After first gc weakHashMap is not empty
>> s04 finalize called
After first gc refQueue has null
>> s02 finalize called

After second gc strongRef is not null
After second gc softRef is not null
After second gc weakRef is null
After second gc weakHashMap is not empty
After second gc refQueue has java.lang.ref.PhantomReference@15d63da

After Finalization strongRef is not null
After Finalization softRef is not null
After Finalization weakRef is null
After Finalization weakHashMap is not empty
After Finalization refQueue has null

Total memory is 14221312
Total memory is 14221312
When mem is low strongRef is not null
When mem is low softRef is not null
When mem is low weakRef is null
When mem is low weakHashMap is not empty
When mem is low refQueue has null

>> s01 finalize called
Not enough memory
tmp1 is null
When not enough memory strongRef is not null
When not enough memory softRef is null
When not enough memory weakRef is null
When not enough memory weakHashMap is empty
When not enough memory refQueue has null

R语言做线性拟合

1、线性拟合

#生成测试数据
x = seq(-5,5,0.1)
y = 3*x^2+6*x+9+rnorm(length(x))*3;

#把x^2用I来标记成一个变量
#然后进行线性拟合
z=lm(y~I(x^2)+x)

#绘制数据点
#及拟合曲线
plot(x,y)
lines(x,fitted(z))

2、局部多项式回归拟合

#生成测试数据
x = seq(-5,5,0.1)
y = 3*x^2+6*x+9+rnorm(length(x))*3;

#局部多项式回归拟合
z=predict(loess(y~x))

#绘制数据点
#及拟合曲线
plot(x,y)
lines(x,z)

#lowes默认使用局部多项式回归拟合
#z1=lowess(x,y)
#lines(z1)

3、非线性最小二乘拟合

#生成测试数据
x = seq(-5,5,0.1)
y = 3*x^2+6*x+9+rnorm(length(x))*3
ds <- data.frame(x=x, y=y)

#进行非线性最小二乘拟合
f=function(x, a, b, c, d) {a+b*x+c*x^2}
z=nls(y~f(x, a, b, c), data=ds, start=list(a=9, b=6, c=3))

#输出拟合结果
summary(z);

#绘制数据点
#及拟合曲线
plot(x,y)
lines(x,fitted(z))

R语言做聚类分析

1、分层聚类

#载入R自带的测试数据
#data(iris)
#attach(iris)
inData = iris[,1:4]

#计算距离矩阵,并绘图
inData.dist = dist(inData)
#inData.dist = dist(inData,method='euclidean')
heatmap(as.matrix(inData.dist), labRow = F, labCol = F)

#进行分层聚类
#并绘图
#inData.hc <- hclust(inData.dist)
#inData.hc <- hclust(inData.dist,method='ward')
plot(inData.hc, labels = FALSE, hang = -1)

#标识聚类结果,结果设为3类
rect.hclust(inData.hc, k = 3)

#将Tree进行分组
inData.groups <- cutree(inData.hc, 3)
#输出结果表格
table(inData.groups, Species)

#进行降维处理
#绘图对比结果
#形状是正确的数据
#颜色为聚类后的数据
mds=cmdscale(inData.dist,k=2,eig=T)
x = mds$points[,1]
y = mds$points[,2]
library(ggplot2)
p=ggplot(data.frame(x,y),aes(x,y))
p+geom_point(size=3,alpha=0.8,aes(colour=factor(inData.groups),shape=iris$Species))

ClusterAnalysisH.png

2、K值聚类

#载入R自带的测试数据
#data(iris)
#attach(iris)
inData = iris[,1:4]

#计算距离矩阵,并绘图
inData.dist = dist(inData)
#inData.dist = dist(inData,method='euclidean')
heatmap(as.matrix(inData.dist), labRow = F, labCol = F)

#进行K值聚类
inData.kc <- kmeans(inData.dist,centers=3)

#绘图对比结果
#形状是正确的数据
#颜色为聚类后的数据
library(ggplot2)
x=inData[c("Sepal.Length")]
y=inData[c("Sepal.Width")]
p=ggplot(data.frame(x,y),aes(x,y))
p+geom_point(size=3,alpha=0.8,aes(colour=factor(inData.kc$cluster),shape=iris$Species))

ClusterAnalysisK.png

编译Wkhtmltopdf

1、首先下载源码

git clone https://github.com/wkhtmltopdf/wkhtmltopdf.git  D:\GitHub\wkhtmltopdf
git clone https://github.com/wkhtmltopdf/qt.git  D:\GitHub\wkhtmltopdf\qt

2、安装下面几个软件
VS2013
Python2.7
ActivePerl
NSIS

3、初始化环境变量

set GIT_HOME=C:\Program Files\Git
set NSIS_HOME=C:\NeoLanguages\NSIS
set PYTHON_HOME=C:\NeoLanguages\Python27_x86
set PERL_HOME=C:\NeoLanguages\Perl
set PATH=%GIT_HOME%\bin;%PYTHON_HOME%;%PERL_HOME%\bin;%NSIS_HOME%;%PATH%
@call C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat x86

4、编译

python scripts\build.py msvc2013-win32

PS:
如果编译失败,一般是由于依赖包的网站无法方位导致的,你懂的。
这个时候,打开build.py文件,找到DEPENDENT_LIBS,自备梯子,下载后直接放到static-build路径下就好了。

posix_memalign函数在Windows下的实现

posix_memalign函数主要用于申请内存时,做内存对齐使用,Windows下对应的函数为_aligned_malloc,但两者的参数有一定区别:

int posix_memalign(void **memptr, size_t alignment, size_t size);
void * _aligned_malloc(size_t size, size_t alignment);

从stackoverflow上,找到了两种实现方式,对于第一种,我只能说,佩服佩服。

1、最简练的实现方式

#define posix_memalign(p, a, s) (((*(p)) = _aligned_malloc((s), (a))), *(p) ?0 :errno)

2、比较稳妥的实现方式

#ifdef _WIN32
static int check_align(size_t align)
{
    for (size_t i = sizeof(void *); i != 0; i *= 2)
    if (align == i)
        return 0;
    return EINVAL;
}

int posix_memalign(void **ptr, size_t align, size_t size)
{
    if (check_align(align))
        return EINVAL;

    int saved_errno = errno;
    void *p = _aligned_malloc(size, align);
    if (p == NULL)
    {
        errno = saved_errno;
        return ENOMEM;
    }

    *ptr = p;
    return 0;
}
#endif