OpenResty学习笔记


这是我在学习OpenResty时记录的一些信息,包含我学习时的代码和记录的函数功能表格。我学习的课程是《OpenResty最佳实践》,感谢来自360的作者moonbingbing。

我的目的是构建一个高性能WAF。我将使用OpenResty与Redis来构建。

OpenResty安装

环境:Centos7.5
OpenResty版本:openresty-1.13.6.2

yum update -y
yum install readline-devel pcre-devel openssl-devel perl gcc automake autoconf libtool make epel-release redis -y
cd Downloads
wget https://openresty.org/download/openresty-1.13.6.2.tar.gz
tar -xzvf openresty-1.13.6.2.tar.gz
cd openresty-1.13.6.2
./configure --with-luajit\
            --without-http_redis2_module \
            --with-http_iconv_module
gmake
gmake install

添加PATH

vim /etc/profile

/etc/profile

export PATH=$PATH:/usr/local/openresty/nginx/sbin

刷新/etc/profile

source /etc/profile

默认安装目录:/usr/local/openresty

HelloWorld

创建工作目录

在当前用户目录下创建openresty-test openresty-test/logs openresty-test/conf文件夹

mkdir ~/openresty-test ~/openresty-test/logs ~/openresty-test/conf

新建配置文件

vim ~/openresty-test/conf/nginx.conf

~/openresty-test/conf/nginx.conf

worker_processes  1;        #nginx worker 数量 
error_log logs/error.log;   #指定错误日志文件路径
events {
    worker_connections 1024;
}   

http {
    server {
        #监听端口
        listen 6699; 
        location / {
            default_type text/html;
            
            content_by_lua_block {
                ngx.say("HelloWorld")
            }   
        }   
    }   
}   

启动Nginx

nginx -p ~/openresty-test

访问localhost:6699

curl http://localhost:6699 -i

输出

HTTP/1.1 200 OK
Server: openresty/1.13.6.2
Date: Tue, 05 Mar 2019 06:18:33 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive

HelloWorld

获取uri参数

openresty给出了两个函数分别用来获取GET和POST信息,ngx.req.get_uri_args()、ngx.req.get_post_args(),要注意的是获取post信息之前一定要先使用ngx.req.read_body()读取 body。

vim ~/openresty-test/conf/nginx.conf

~/openresty-test/conf/nginx.conf

worker_processes  1;        #nginx worker 数量 
error_log logs/error.log;   #指定错误日志文件路径
events {
    worker_connections 1024;
}   

http {
    server {
       listen    80;
       server_name  localhost;
    
       location /print_param {
           content_by_lua_block {
               local arg = ngx.req.get_uri_args()
               for k,v in pairs(arg) do
                   ngx.say("[GET ] key:", k, " v:", v)
               end
    
               ngx.req.read_body() -- 解析 body 参数之前一定要先读取 body
               local arg = ngx.req.get_post_args()
               for k,v in pairs(arg) do
                   ngx.say("[POST] key:", k, " v:", v)
               end
           }
       }
    }
}  

启动Nginx

nginx -p ~/openresty-test

访问localhost:6699

curl 'localhost/print_param?a=1&b=2%26' -d 'c=3&d=4%26'

输出

[GET ] key:b v:2&
[GET ] key:a v:1
[POST] key:d v:4&
[POST] key:c v:3

获取请求Body

openresty给出了一个函数用来获取请求Body,要注意的是获取请求Body之前要把默认读取Body选项打开。

vim ~/openresty-test/conf/nginx.conf

~/openresty-test/conf/nginx.conf

worker_processes  1;        #nginx worker 数量 
error_log logs/error.log;   #指定错误日志文件路径
events {
    worker_connections 1024;
}   

http {
    server {
        listen    80;
        
        # 默认读取 body
        lua_need_request_body on;
        
        location /test {
            content_by_lua_block {
                local data = ngx.req.get_body_data()
                ngx.say("hello ", data)
            }
        }
    }
}  

启动Nginx

nginx -p ~/openresty-test

访问localhost:6699

curl localhost/test -d 'xiaoming'

输出

hello xiaoming

Redis

Redis设置密码与访问

config set requirepass test123

访问

local redis = require "resty.redis"
local red = redis:new()

red:set_timeout(1000) -- 1 sec

local ok, err = red:connect("127.0.0.1", 6379)
if not ok then
    ngx.say("failed to connect: ", err)
    return
end

local count
count, err = red:get_reused_times()
if 0 == count then
    ok, err = red:auth("password")
    if not ok then
        ngx.say("failed to auth: ", err)
        return
    end
elseif err then
    ngx.say("failed to get reused times: ", err)
        return
end

抄一个redis类。以备后用。一般用不上。
resty/redis_iresty.lua

local redis_c = require "resty.redis"


local ok, new_tab = pcall(require, "table.new")
if not ok or type(new_tab) ~= "function" then
    new_tab = function (narr, nrec) return {} end
end


local _M = new_tab(0, 155)
_M._VERSION = '0.01'


local commands = {
    "append",            "auth",              "bgrewriteaof",
    "bgsave",            "bitcount",          "bitop",
    "blpop",             "brpop",
    "brpoplpush",        "client",            "config",
    "dbsize",
    "debug",             "decr",              "decrby",
    "del",               "discard",           "dump",
    "echo",
    "eval",              "exec",              "exists",
    "expire",            "expireat",          "flushall",
    "flushdb",           "get",               "getbit",
    "getrange",          "getset",            "hdel",
    "hexists",           "hget",              "hgetall",
    "hincrby",           "hincrbyfloat",      "hkeys",
    "hlen",
    "hmget",              "hmset",      "hscan",
    "hset",
    "hsetnx",            "hvals",             "incr",
    "incrby",            "incrbyfloat",       "info",
    "keys",
    "lastsave",          "lindex",            "linsert",
    "llen",              "lpop",              "lpush",
    "lpushx",            "lrange",            "lrem",
    "lset",              "ltrim",             "mget",
    "migrate",
    "monitor",           "move",              "mset",
    "msetnx",            "multi",             "object",
    "persist",           "pexpire",           "pexpireat",
    "ping",              "psetex",            "psubscribe",
    "pttl",
    "publish",      --[[ "punsubscribe", ]]   "pubsub",
    "quit",
    "randomkey",         "rename",            "renamenx",
    "restore",
    "rpop",              "rpoplpush",         "rpush",
    "rpushx",            "sadd",              "save",
    "scan",              "scard",             "script",
    "sdiff",             "sdiffstore",
    "select",            "set",               "setbit",
    "setex",             "setnx",             "setrange",
    "shutdown",          "sinter",            "sinterstore",
    "sismember",         "slaveof",           "slowlog",
    "smembers",          "smove",             "sort",
    "spop",              "srandmember",       "srem",
    "sscan",
    "strlen",       --[[ "subscribe",  ]]     "sunion",
    "sunionstore",       "sync",              "time",
    "ttl",
    "type",         --[[ "unsubscribe", ]]    "unwatch",
    "watch",             "zadd",              "zcard",
    "zcount",            "zincrby",           "zinterstore",
    "zrange",            "zrangebyscore",     "zrank",
    "zrem",              "zremrangebyrank",   "zremrangebyscore",
    "zrevrange",         "zrevrangebyscore",  "zrevrank",
    "zscan",
    "zscore",            "zunionstore",       "evalsha"
}


local mt = { __index = _M }


local function is_redis_null( res )
    if type(res) == "table" then
        for k,v in pairs(res) do
            if v ~= ngx.null then
                return false
            end
        end
        return true
    elseif res == ngx.null then
        return true
    elseif res == nil then
        return true
    end

    return false
end


-- change connect address as you need
function _M.connect_mod( self, redis )
    redis:set_timeout(self.timeout)
    return redis:connect("127.0.0.1", 6379)
end


function _M.set_keepalive_mod( redis )
    -- put it into the connection pool of size 100, with 60 seconds max idle time
    return redis:set_keepalive(60000, 1000)
end


function _M.init_pipeline( self )
    self._reqs = {}
end


function _M.commit_pipeline( self )
    local reqs = self._reqs

    if nil == reqs or 0 == #reqs then
        return {}, "no pipeline"
    else
        self._reqs = nil
    end

    local redis, err = redis_c:new()
    if not redis then
        return nil, err
    end

    local ok, err = self:connect_mod(redis)
    if not ok then
        return {}, err
    end

    redis:init_pipeline()
    for _, vals in ipairs(reqs) do
        local fun = redis[vals[1]]
        table.remove(vals , 1)

        fun(redis, unpack(vals))
    end

    local results, err = redis:commit_pipeline()
    if not results or err then
        return {}, err
    end

    if is_redis_null(results) then
        results = {}
        ngx.log(ngx.WARN, "is null")
    end
    -- table.remove (results , 1)

    self.set_keepalive_mod(redis)

    for i,value in ipairs(results) do
        if is_redis_null(value) then
            results[i] = nil
        end
    end

    return results, err
end


function _M.subscribe( self, channel )
    local redis, err = redis_c:new()
    if not redis then
        return nil, err
    end

    local ok, err = self:connect_mod(redis)
    if not ok or err then
        return nil, err
    end

    local res, err = redis:subscribe(channel)
    if not res then
        return nil, err
    end

    local function do_read_func ( do_read )
        if do_read == nil or do_read == true then
            res, err = redis:read_reply()
            if not res then
                return nil, err
            end
            return res
        end

        redis:unsubscribe(channel)
        self.set_keepalive_mod(redis)
        return
    end

    return do_read_func
end


local function do_command(self, cmd, ... )
    if self._reqs then
        table.insert(self._reqs, {cmd, ...})
        return
    end

    local redis, err = redis_c:new()
    if not redis then
        return nil, err
    end

    local ok, err = self:connect_mod(redis)
    if not ok or err then
        return nil, err
    end

    local fun = redis[cmd]
    local result, err = fun(redis, ...)
    if not result or err then
        -- ngx.log(ngx.ERR, "pipeline result:", result, " err:", err)
        return nil, err
    end

    if is_redis_null(result) then
        result = nil
    end

    self.set_keepalive_mod(redis)

    return result, err
end


for i = 1, #commands do
    local cmd = commands[i]
    _M[cmd] =
            function (self, ...)
                return do_command(self, cmd, ...)
            end
end


function _M.new(self, opts)
    opts = opts or {}
    local timeout = (opts.timeout and opts.timeout * 1000) or 1000
    local db_index= opts.db_index or 0

    return setmetatable({
            timeout = timeout,
            db_index = db_index,
            _reqs = nil }, mt)
end


return _M

使用示例:

local redis = require "resty.redis_iresty"
local red = redis:new()

local ok, err = red:set("dog", "an animal")
if not ok then
    ngx.say("failed to set dog: ", err)
    return
end

ngx.say("set result: ", ok)

正则表达式

Lua 中正则表达式的性能不如 ngx.re.* 中的正则表达式优秀。
Lua 中的正则表达式并不符合 POSIX 规范,而 ngx.re.* 中实现的是标准的 POSIX 规范。
Lua 中的正则表达式与 Nginx 中的正则表达式相比,有 5% - 15% 的性能损失,而且 Lua 将表达式编译成 Pattern 之后,并不会将 Pattern 缓存,而是每此使用都重新编译一遍,潜在地降低了性能。
ngx.re.* 中的正则表达式可以通过参数缓存编译过后的 Pattern,不会有类似的性能损失。
ngx.re.* 中的 o 选项,指明该参数,被编译的 Pattern 将会在工作进程中缓存,并且被当前工作进程的每次请求所共享。Pattern 缓存的上限值通过 lua_regex_cache_max_entries 来修改。
ngx.re.* 中的 j 选项,指明该参数,如果使用的 PCRE 库支持 JIT,OpenResty 会在编译 Pattern 时启用 JIT。
总之,只用ngx.re. 就对了,ngx.re. 使用的是一般的正则规范,Lua中的那一套正则就不用了。

location /test {
    content_by_lua_block {
        local regex = [[\d+]]

        -- 参数 "j" 启用 JIT 编译,参数 "o" 是开启缓存必须的
        local m = ngx.re.match("hello, 1234", regex, "jo")
        if m then
            ngx.say(m[0])
        else
            ngx.say("not matched!")
        end
    }
}

Nginx下lua处理阶段

nginx下lua处理有以下阶段

  1. init_by_lua http
    当 nginx master 进程在加载 nginx 配置文件时运行指定的 lua 脚本,通常用来注册 lua 的全局变量或在服务器启动时预加载 lua 模块。
  2. set_by_lua server, server if, location, location if
    阶段:rewrite
  3. rewrite_by_lua http, server, location, location if
    阶段:rewrite tail

作为rewrite阶段的处理,为每个请求执行指定的lua代码。

  1. access_by_lua http, server, location, location if
    阶段:access tail

为每个请求在访问阶段的调用lua脚本进行处理。主要用于访问控制,能收集到大部分的变量。这条指令运行于nginx access阶段的末尾,因此总是在 allow 和 deny 这样的指令之后运行,虽然它们同属 access 阶段。

  1. content_by_lua location, location if
    阶段:content

作为“content handler”为每个请求执行lua代码,为请求者输出响应内容。此阶段是所有请求处理阶段中最为重要的一个,运行在这个阶段的配置指令一般都肩负着生成内容(content)并输出HTTP响应。

  1. header_filter_by_lua http, server, location, location if
    一般用来设置cookie和headers,在该阶段不能使用如下几个API:
  2. body_filter_by_lua http, server, location, location if
    阶段:output-body-filter

一般会在一次请求中被调用多次, 因为这是实现基于 HTTP 1.1 chunked 编码的所谓“流式输出”的。

  1. log_by_lua http, server, location, location if
    阶段:log

在log阶段调用指定的lua脚本,并不会替换access log,而是在那之后进行调用。该阶段总是运行在请求结束的时候,用于请求的后续操作,如在共享内存中进行统计数据,如果要高精确的数据统计,应该使用body_filter_by_lua。

Nginx函数与变量

ngx.shared.DICT

ngx.shared.DICT意思是ngx.shared.任意字典名。ngx.shared.DICT里存储一个全局字典,能够给这个字典设置值和值的过期时间。
例如:ngx.shared.limit

ngx.shared.DICT.set

ngx.shared.DICT.set()是一个函数,能够设置ngx.shared.DICT的值。
例如:ngx.shared.limit:set(token,1,3))
意思是在limit字典中新增token字段,值为1,3秒有效期,3秒后值为nil。

ngx.shared.DICT.incr

ngx.shared.DICT.incr()是一个函数,能给字典中的字段自增。
例如:ngx.shared.limit:incr(token,1)
意思是给token自增1。

ngx.shared.DICT.get

ngx.shared.DICT.incr()是一个函数,能得到字典中字段的值。
例如:r,_=ngx.shared.limit:(token)
注意,一定要有两个值的返回,使用_忽略第二个值。第二个值一般为空。


以上即是我学习OpenResty的笔记,我的邮箱是wubo@wubo.net.cn,欢迎和我讨论。

声明:物博网|版权所有,违者必究|如未注明,均为原创|本网站采用BY-NC-SA协议进行授权

转载:转载请注明原文链接 - OpenResty学习笔记


喜欢安全与WEB开发