Wireshark (previously Ethereal) is the best open source protocol dissector/analyzer. You can analyze an incredible amount of protocols, not only Internet ones, but every stream based protocols. Moreover you can add your own filters/dissectors written either in C or in Lua. Nevertheless, the documentation on the net concerning Lua dissectors is light and sparse. It's been hard for me to make something that works even if it's, at the end, not really complicated. I'll try to explain the basis of Lua dissectors.

1) Installation

You need to have a wireshark that supports Lua support (wireshark -v). After that, create or edit ~/.wireshark/init.lua. To load a new plugin, just type

dofile("mydissector.lua")

Create the new file ~/.wireshark/mydissector.lua

2) Post Dissector

There are three types of dissectors :

• Dissector : you add your own protocol
• Chained dissector : you add new fields to an existing protocol
• Post dissector : you interact after all packets has been parsed

An example for each of one can be found here. I'll describe a post dissector, but other types of dissectors has pretty the same format.

Tip 1
If you want to display someting on the console, just do

print(something)

Tip 2
If the base array is not defined, add this to ~/.wireshark/init.lua

-- Display Bases
 base = {
    ["NONE"] = 0,
    ["DEC"] = 1,
    ["HEX"] = 2,
    ["OCT"] = 3,
    ["DEC_HEX"] = 4,
    ["HEX_DEC"] = 5,
}

First, you need to define your protocol : Proto(<internal name>, <displayed name>)

p_dummyproto = Proto("dummyproto","DummyProto")

Then, define your fields : ProtoField.TYPE(<internal name>, <displayed name>, [base], [values], [mask])
TYPE are defined here

-- Simple field without value
local f_X =  ProtoField.uint16("dummyproto.f_X","Field X")
-- Simple field displayed in hex format
local f_Y =  ProtoField.uint8("dummyproto.f_Y","Field Y", base.HEX)
-- Field with precomputed values and bitfield
local VALS_ZZ    = {[0] = "Single", [1] = "Dual"}
local f_Z =  ProtoField.uint8("dummyproto.f_Z","Field Z", base.HEX, VALS_ZZ, 0x3)

Third step is to register each field

p_dummyproto.fields = {f_X, f_Y, f_Z}

After that, the big part : protocol dissection. Fields are organized as a tree. You have to parse each byte (or range of bytes) in the given buffer and append your fields. Be careful : objects returned by :add function has userdata type and cannot be directly manipulated.

function p_dummyproto.dissector(buffer,pinfo,tree)
   -- Access to another field
   local f_udp_port = Field.new("udp.port")

   -- If it exists and has the right value
   if f_udp_port and tostring(f_udp_port) == tostring(5555) then
     -- Add our protocol dissection with data in buffer[17, 17+14]
     local subtree = tree:add(p_dummyproto, buffer(17,14))

     -- Add a subtree to our root for the first two bytes
     local t = subtree:add(f_X, buffer(17, 2))

     -- Add a sub subtree
     local t2 = t:add(f_Y, buffer(17, 1))

     -- Parse sub data
     parse_data(t, buffer, 18)
   end
end

function parse_data(tree, buffer, start)
     -- Wireshark integrate bitop from luajit http://bitop.luajit.org/
     field_1 = buffer(start, 1):uint()
     field_1 = bit.band(field_1, 0x3)

     -- You can also append free text information to current field
     if field_1 < 16 then
         tree:append_text(" field information")
     end
end

Finally register your post dissector

register_postdissector(p_dummyproto)

A complete example can be found here. It's a full reimplementation of ARP protocol dissector in Lua.