<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet href="http://blog.xole.net/rss/style.css" type="text/css"?>
<rdf:RDF xmlns="http://purl.org/rss/1.0/"
         xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:content="http://purl.org/rss/1.0/modules/content/"
         xmlns:dc="http://purl.org/dc/elements/1.1/"
         xml:lang="ja">
<channel rdf:about="http://blog.xole.net/rss/1.0.php?id=738">
<title>ハタさんのブログ(復刻版)</title>
<link>http://blog.xole.net/index.php</link>
<dc:date>2010-01-11T23:39:32+09:00</dc:date>
<description>
ハタさんのブログ(復刻版) - RSS (RDF Site Summary).
</description>
<items>
<rdf:Seq>
<rdf:li rdf:resource="http://blog.xole.net/article.php?id=738" />
</rdf:Seq>
</items>
</channel>
<item>
<title>JavaCC で memcache text protocol の BNF(と、なんちゃってmemcache互換サーバ)</title>
<link>http://blog.xole.net/article.php?id=738</link>
<dc:date>2010-01-11T23:39:32+09:00</dc:date>
<description>ANTLRのやつがあったけど、JavaCCのがみつからなかったので、でっちあげた
via - http://harward.us/~nharward/antlr/memcached_protocol.g


できあがったのは、こんな...</description>
<content:encoded>
<![CDATA[
<p>ANTLRのやつがあったけど、JavaCCのがみつからなかったので、でっちあげた<br />
via - <a href="http://harward.us/~nharward/antlr/memcached_protocol.g">http://harward.us/~nharward/antlr/memcached_protocol.g</a>
</p>

<p>できあがったのは、こんな感じ</p>

<pre class="javascript">SKIP: {
    <span class="string" >" "</span> | <span class="string" >"\t"</span> | <span class="string" >"\r"</span> | <span class="string" >"\n"</span>
}
TOKEN: {
    &lt; NUMBER: [<span class="string" >"1"</span>-<span class="string" >"9"</span>] ([<span class="string" >"0"</span>-<span class="string" >"9"</span>])* | <span class="string" >"0"</span> &gt;
  | &lt; FLAGS: &lt; NUMBER &gt; &gt;
  | &lt; TIME: &lt; NUMBER &gt;  &gt;
  | &lt; LENGTH: &lt; NUMBER &gt; &gt;
  | &lt; CREMENT_VALUE: &lt; NUMBER &gt; &gt;
  | &lt; CAS_UNIQUE: &lt; NUMBER &gt; &gt;
}
TOKEN: {
  &lt; SET_STATEMENT: <span class="string" >"set"</span> &gt;
  | &lt; ADD_STATEMENT: <span class="string" >"add"</span> &gt;
  | &lt; REPLACE_STATEMENT: <span class="string" >"replace"</span> &gt;
  | &lt; APPEND_STATEMENT: <span class="string" >"append"</span> &gt;
  | &lt; PREPEND_STATEMENT: <span class="string" >"prepend"</span> &gt;
  | &lt; CAS_STATEMENT: <span class="string" >"cas"</span> &gt;
  | &lt; STORAGE_STATEMENT:
        &lt; SET_STATEMENT &gt;
        | &lt; ADD_STATEMENT &gt;
        | &lt; REPLACE_STATEMENT &gt;
        | &lt; APPEND_STATEMENT &gt;
        | &lt; PREPEND_STATEMENT &gt;
    &gt;
  | &lt; STORAGE_COMMAND:
        (
          &lt; STORAGE_STATEMENT &gt; &lt; KEY &gt; &lt; FLAGS &gt; &lt; TIME &gt; &lt; LENGTH &gt;
          | &lt; CAS_STATEMENT &gt; &lt; KEY &gt; &lt; FLAGS &gt; &lt; TIME &gt; &lt; LENGTH &gt; &lt; CAS_UNIQUE &gt;
        )
        (&lt; NOREPLY &gt;)?
    &gt;
}
TOKEN: {
  &lt; RETRIEVAL_STATEMENT: <span class="string" >"get"</span> | <span class="string" >"gets"</span> &gt;
  | &lt; RETRIEVAL_COMMAND:
        &lt; RETRIEVAL_STATEMENT &gt; &lt; KEY &gt;
    &gt;
}
TOKEN: {
  &lt; DELETE_STATEMENT: <span class="string" >"delete"</span> &gt;
  | &lt; DELETE_COMMAND:
        &lt; DELETE_STATEMENT &gt; &lt; KEY &gt; (&lt; TIME &gt;)? (&lt; NOREPLY &gt;)?
    &gt;
}
TOKEN: {
  &lt; INCREMENT_STATEMENT: <span class="string" >"incr"</span> &gt;
  | &lt; INCREMENT_COMMAND:
        &lt; INCREMENT_STATEMENT &gt; &lt; KEY &gt; &lt; CREMENT_VALUE &gt; (&lt; NOREPLY &gt;)?
    &gt;
}
TOKEN: {
  &lt; DECREMENT_STATEMENT: <span class="string" >"decr"</span> &gt;
  | &lt; DECREMENT_COMMAND:
        &lt; DECREMENT_STATEMENT &gt; &lt; KEY &gt; &lt; CREMENT_VALUE &gt; (&lt; NOREPLY &gt;)?
    &gt;
}
TOKEN: {
  &lt; STATISTICS_STATEMENT: <span class="string" >"STAT"</span> &gt;
  | &lt; STATISTICS_OPTION: <span class="string" >"items"</span> | <span class="string" >"slabs"</span> | <span class="string" >"sizes"</span> &gt;
  | &lt; STATISTICS_COMMAND:
        &lt; STATISTICS_STATEMENT &gt; (&lt; STATISTICS_OPTION &gt;)?
    &gt;
}
TOKEN: {
  &lt; FLUSH_STATEMENT: <span class="string" >"flush_all"</span> &gt;
  | &lt; FLUSH_COMMAND:
        &lt; FLUSH_STATEMENT &gt; (&lt; TIME &gt;)? (&lt; NOREPLY &gt;)?
    &gt;
}
TOKEN: {
  &lt; VERSION_STATEMENT: <span class="string" >"version"</span> &gt;
  | &lt; VERSION_COMMAND:
        &lt; VERSION_STATEMENT &gt;
    &gt;
}
TOKEN: {
    &lt; NOREPLY: <span class="string" >"noreply"</span> &gt;
}
<span class="comment" >// last match</span>
TOKEN: {
    &lt; KEY: (~[<span class="string" >" "</span>, <span class="string" >"\r"</span>,<span class="string" >"\n"</span>])+ &gt;
}</pre>

<p>少し、数値まわりのToken(TIME, LENGTHとか)が適当すぎるかな。</p>

<p>んで、これにテキトーなNodeをparseしてあげてみる</p>

<pre class="java">Command Command():
{
  Command command;
}
{
  (
    command = RetrievalCommand()
  | command = StorageCommand()
  | command = DeleteCommand()
  | command = VersionCommand()
  )
  {
    <span class="keyword" >return</span> command;
  }
}
StorageCommand StorageCommand():
{
  StorageCommand command;
  String key;
  Long flags = 0L;
  Long time = 0L;
  Long length = 0L;
  Boolean noreply = Boolean.FALSE;
}
{
  command = createStorageCommand()
  key = Key()
  flags = Flags()
  time = Time()
  length = Length()
  noreply = Noreply()
  {
    command.setNode(jjtThis);
    command.setKey(key);
    command.setFlags(flags);
    command.setExpTime(time);
    command.setLength(length);
    command.setNoreply(noreply);
    <span class="keyword" >return</span> command;
  }
}
StorageCommand createStorageCommand():
{}
{
  (
    &lt; SET_STATEMENT &gt;
    {
      <span class="keyword" >return</span> <span class="keyword" >new</span> SetCommand();
    }
    | &lt; ADD_STATEMENT &gt;
    {
      <span class="keyword" >return</span> <span class="keyword" >new</span> AddCommand();
    }
    | &lt; REPLACE_STATEMENT &gt;
    {
      <span class="keyword" >return</span> <span class="keyword" >new</span> ReplaceCommand();
    }
    | &lt; APPEND_STATEMENT &gt;
    {
      <span class="keyword" >return</span> <span class="keyword" >new</span> AppendCommand();
    }
    | &lt; PREPEND_STATEMENT &gt;
    {
      <span class="keyword" >return</span> <span class="keyword" >new</span> PrependCommand();
    }
  )
}

RetrievalCommand RetrievalCommand():
{
  RetrievalCommand command = <span class="keyword" >new</span> RetrievalCommand();
  String key;
}
{
  &lt; RETRIEVAL_STATEMENT &gt;
  (
    key = Key()
    {
      command.addKey(key);
    }
  )+
  {
    command.setNode(jjtThis);
    <span class="keyword" >return</span> command;
  }
}
DeleteCommand DeleteCommand():
{
  DeleteCommand command = <span class="keyword" >new</span> DeleteCommand();
  String key;
  Long time = 0L;
  Boolean noreply = Boolean.FALSE;
}
{
  &lt; DELETE_STATEMENT &gt;
  key = Key()
  time = Time()
  noreply = Noreply()
  {
    command.setNode(jjtThis);
    command.setKey(key);
    command.setExpTime(time);
    command.setNoreply(noreply);
    <span class="keyword" >return</span> command;
  }
}
VersionCommand VersionCommand():
{}
{
  &lt; VERSION_STATEMENT &gt;
  {
    VersionCommand command = <span class="keyword" >new</span> VersionCommand();
    command.setNode(jjtThis);
    <span class="keyword" >return</span> command;
  }
}

String Key():
{ Token key; }
{
  key = &lt; KEY &gt;
  {
    <span class="keyword" >return</span> key.image;
  }
}

Long Flags():
{ Token flags; }
{
  flags = &lt; NUMBER &gt;
  {
    <span class="keyword" >return</span> Long.valueOf(flags.image);
  }
}

Long Time():
{ Token time; }
{
  time = &lt; NUMBER &gt;
  {
    <span class="keyword" >return</span> Long.valueOf(time.image);
  }
}

Long Length():
{ Token length; }
{
  length = &lt; NUMBER &gt;
  {
    <span class="keyword" >return</span> Long.valueOf(length.image);
  }
}

Boolean Noreply():
{ Boolean noreply = Boolean.FALSE; }
{
  [&lt; NOREPLY &gt;{noreply = Boolean.TRUE;}]
  {
    <span class="keyword" >return</span> noreply;
  }
}</pre>

<p>これに、適当なコードを投げてあげると</p>

<pre class="java">{
    StringReader reader = <span class="keyword" >new</span> StringReader(<span class="string" >"get hoge\r\n"</span>);
    MemcacheParser parser = <span class="keyword" >new</span> MemcacheParser(reader);
    <span class="keyword" >try</span> {
        parser.Command();
    } <span class="keyword" >catch</span> (ParseException e) {
        e.printStackTrace();
    }
}
{
    StringReader reader = <span class="keyword" >new</span> StringReader(<span class="string" >"gets hoge foo\r\n"</span>);
    MemcacheParser parser = <span class="keyword" >new</span> MemcacheParser(reader);
    <span class="keyword" >try</span> {
        parser.Command();
    } <span class="keyword" >catch</span> (ParseException e) {
        e.printStackTrace();
    }
}
{
    StringReader reader = <span class="keyword" >new</span> StringReader(<span class="string" >"set xyzkey 0 0 6\r\n"</span>);
    MemcacheParser parser = <span class="keyword" >new</span> MemcacheParser(reader);
    <span class="keyword" >try</span> {
        parser.Command();
    } <span class="keyword" >catch</span> (ParseException e) {
        e.printStackTrace();
    }
}</pre>

<pre class="java">Call:   Command
  Call:   RetrievalCommand
    Consumed token: &lt;&lt;RETRIEVAL_STATEMENT&gt;: <span class="string" >"get"</span> at line <span class="number" >1</span> column <span class="number" >1</span>&gt;
    Call:   Key
      Consumed token: &lt;&lt;KEY&gt;: <span class="string" >"hoge"</span> at line <span class="number" >1</span> column <span class="number" >5</span>&gt;
    Return: Key
  Return: RetrievalCommand
Return: Command
Call:   Command
  Call:   RetrievalCommand
    Consumed token: &lt;&lt;RETRIEVAL_STATEMENT&gt;: <span class="string" >"gets"</span> at line <span class="number" >1</span> column <span class="number" >1</span>&gt;
    Call:   Key
      Consumed token: &lt;&lt;KEY&gt;: <span class="string" >"hoge"</span> at line <span class="number" >1</span> column <span class="number" >6</span>&gt;
    Return: Key
    Call:   Key
      Consumed token: &lt;&lt;KEY&gt;: <span class="string" >"foo"</span> at line <span class="number" >1</span> column <span class="number" >11</span>&gt;
    Return: Key
  Return: RetrievalCommand
Return: Command
Call:   Command
  Call:   StorageCommand
    Call:   createStorageCommand
      Consumed token: &lt;<span class="string" >"set"</span> at line <span class="number" >1</span> column <span class="number" >1</span>&gt;
    Return: createStorageCommand
    Call:   Key
      Consumed token: &lt;&lt;KEY&gt;: <span class="string" >"xyzkey"</span> at line <span class="number" >1</span> column <span class="number" >5</span>&gt;
    Return: Key
    Call:   Flags
      Consumed token: &lt;&lt;NUMBER&gt;: <span class="string" >"0"</span> at line <span class="number" >1</span> column <span class="number" >12</span>&gt;
    Return: Flags
    Call:   Time
      Consumed token: &lt;&lt;NUMBER&gt;: <span class="string" >"0"</span> at line <span class="number" >1</span> column <span class="number" >14</span>&gt;
    Return: Time
    Call:   Length
      Consumed token: &lt;&lt;NUMBER&gt;: <span class="string" >"6"</span> at line <span class="number" >1</span> column <span class="number" >16</span>&gt;
    Return: Length
    Call:   Noreply
    Return: Noreply
  Return: StorageCommand
Return: Command</pre>

<p>と、こんな感じになる。<br />
&lt;NUMBER&gt;とか、ホント、マジメにtokenが書けてないですね。。</p>

<p>ここまでできたので、set と get しかない、memcached 互換ものをでっち上げてみた</p>

<pre class="java">
<span class="keyword" >public</span> <span class="keyword" >class</span> Server <span class="keyword" >extends</span> Thread {
    
    <span class="keyword" >protected</span> <span class="keyword" >final</span> BlockingQueue&lt;Socket&gt; accept = <span class="keyword" >new</span> LinkedBlockingQueue&lt;Socket&gt;();
    
    <span class="keyword" >protected</span> <span class="keyword" >final</span> Cache&lt;String, String&gt; cache = <span class="keyword" >new</span> LRUCache&lt;String, String&gt;();

    <span class="keyword" >protected</span> <span class="keyword" >final</span> ExecutorService acceptPool;
    
    <span class="keyword" >protected</span> <span class="keyword" >final</span> <span class="keyword" >int</span> port;
    
    <span class="keyword" >protected</span> <span class="keyword" >final</span> <span class="keyword" >int</span> maxConnection;
    
    <span class="keyword" >public</span> Server(<span class="keyword" >final</span> <span class="keyword" >int</span> port, <span class="keyword" >final</span> <span class="keyword" >int</span> maxConnection){
        <span class="keyword" >this</span>.port = port;
        <span class="keyword" >this</span>.maxConnection = maxConnection;
        <span class="keyword" >this</span>.acceptPool = Executors.newFixedThreadPool(maxConnection);
    }
    
    <span class="keyword" >public</span> <span class="keyword" >static</span> <span class="keyword" >void</span> main(String...args){
        Server s = <span class="keyword" >new</span> Server(<span class="number" >12221</span>, <span class="number" >32</span>);
        s.start();
        
        <span class="keyword" >while</span>(s.isAlive()){
            <span class="keyword" >try</span> {
                TimeUnit.MICROSECONDS.sleep(<span class="number" >10</span>);
            } <span class="keyword" >catch</span>(InterruptedException e){}
        }
    }

    <span class="keyword" >public</span> <span class="keyword" >void</span> run(){
        <span class="keyword" >try</span> {
            ServerSocketFactory factory = ServerSocketFactory.getDefault();
            ServerSocket socket = factory.createServerSocket(port, maxConnection);
            socket.setReuseAddress(<span class="keyword" >true</span>);
            
            <span class="keyword" >while</span>(!socket.isClosed()){
                <span class="keyword" >final</span> Socket accept = socket.accept();
                acceptPool.execute(<span class="keyword" >new</span> AcceptHandler(accept));
            }
        } <span class="keyword" >catch</span> (UnknownHostException e) {
            e.printStackTrace();
        } <span class="keyword" >catch</span> (IOException e) {
            e.printStackTrace();
        }
    }
    
    <span class="keyword" >protected</span> <span class="keyword" >class</span> AcceptHandler <span class="keyword" >implements</span> Runnable {
        <span class="keyword" >private</span> <span class="keyword" >final</span> Socket socket;
        <span class="keyword" >public</span> AcceptHandler(<span class="keyword" >final</span> Socket socket){
            <span class="keyword" >this</span>.socket = socket;
        }
        <span class="keyword" >public</span> <span class="keyword" >void</span> run(){
            <span class="keyword" >try</span> {
                <span class="keyword" >final</span> InputStream in = socket.getInputStream();
                <span class="keyword" >final</span> OutputStream out = socket.getOutputStream();
                
                <span class="keyword" >final</span> BufferedReader reader = <span class="keyword" >new</span> BufferedReader(<span class="keyword" >new</span> InputStreamReader(in));
                <span class="keyword" >final</span> DataOutputStream writer = <span class="keyword" >new</span> DataOutputStream(out);
                <span class="keyword" >final</span> CommandWorker worker = <span class="keyword" >new</span> CommandWorker(reader);
                <span class="keyword" >while</span>(!socket.isClosed()){
                    <span class="keyword" >if</span>(!worker.prepare()){
                        <span class="keyword" >break</span>;
                    }
                    Return r = worker.call();
                    writer.writeBytes(r.renderMessage());
                }
            } <span class="keyword" >catch</span>(IOException e){
                e.printStackTrace();
            } <span class="keyword" >finally</span> {
                <span class="keyword" >try</span> {
                    socket.close();
                } <span class="keyword" >catch</span>(IOException e){
                    <span class="comment" >// nop</span>
                }
            }
        }
    }
    <span class="keyword" >private</span> <span class="keyword" >class</span> CommandWorker <span class="keyword" >implements</span> CommandVisitor {
        <span class="keyword" >private</span> <span class="keyword" >final</span> BufferedReader reader;
        <span class="keyword" >private</span> String currentLine;
        <span class="keyword" >public</span> CommandWorker(<span class="keyword" >final</span> BufferedReader reader) {
            <span class="keyword" >this</span>.reader = reader;
        }
        
        <span class="keyword" >public</span> <span class="keyword" >boolean</span> prepare(){
            <span class="keyword" >try</span> {
                String line = reader.readLine();
                <span class="keyword" >if</span>(<span class="keyword" >null</span> == line){
                    <span class="keyword" >return</span> <span class="keyword" >false</span>;
                }
                currentLine = line;
                <span class="keyword" >return</span> <span class="keyword" >true</span>;
            } <span class="keyword" >catch</span>(IOException e){
                <span class="keyword" >return</span> <span class="keyword" >false</span>;
            }
        }
        
        <span class="keyword" >public</span> Return call() {
            <span class="keyword" >try</span> {
                <span class="keyword" >final</span> StringReader r = <span class="keyword" >new</span> StringReader(currentLine);
                <span class="keyword" >final</span> MemcacheParser parser = <span class="keyword" >new</span> MemcacheParser(r);
                
                Command command = parser.Command();
                <span class="keyword" >return</span> command.accept(<span class="keyword" >this</span>, <span class="keyword" >null</span>);
            } <span class="keyword" >catch</span>(ParseException e){
                <span class="keyword" >return</span> <span class="keyword" >new</span> Return(ResponseType.ERROR);
            }
        }
        
        <span class="keyword" >public</span> Return visit(Command command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    
        <span class="keyword" >public</span> Return visit(AddCommand command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    
        <span class="keyword" >public</span> Return visit(AppendCommand command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    
        <span class="keyword" >public</span> Return visit(CasCommand command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    
        <span class="keyword" >public</span> Return visit(DeleteCommand command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    
        <span class="keyword" >public</span> Return visit(PrependCommand command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    
        <span class="keyword" >public</span> Return visit(ReplaceCommand command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    
        <span class="keyword" >public</span> Return visit(RetrievalCommand command, Parameter parameter) {
            String value = cache.get(command.getKeys().get(<span class="number" >0</span>));
            <span class="keyword" >if</span>(<span class="keyword" >null</span> == value){
                <span class="keyword" >return</span> <span class="keyword" >new</span> Return(ResponseType.END);
            }
            <span class="keyword" >return</span> <span class="keyword" >new</span> Return(ResponseType.SEND_VALUE,
                command.getKeys().get(<span class="number" >0</span>), <span class="number" >0</span>, value.length(),
                value
            );
        }
    
        <span class="keyword" >public</span> Return visit(SetCommand command, Parameter parameter) {
            <span class="keyword" >try</span> {
                <span class="keyword" >final</span> String nextLine = reader.readLine();
                cache.put(command.getKey(), nextLine, command.getExpTime().longValue());
                <span class="keyword" >return</span> <span class="keyword" >new</span> Return(ResponseType.STORED);
            } <span class="keyword" >catch</span>(IOException e){
                e.printStackTrace();
                <span class="keyword" >return</span> <span class="keyword" >new</span> Return(ResponseType.ERROR);
            }
        }
    
        <span class="keyword" >public</span> Return visit(VersionCommand command, Parameter parameter) {
            <span class="keyword" >return</span> <span class="keyword" >null</span>;
        }
    }
}</pre>

<p>見事に、setとgetしか実装してません。しかもハンドリングは少し適当。</p>

<p>ということで、これと(java-lang)、memcached(c-lang)で比較してみた。</p>

<pre class="java">$target = array(
    array(<span class="string" >'host'</span> =&gt; <span class="string" >'localhost'</span>, <span class="string" >'port'</span> =&gt; <span class="number" >11211</span>),
    array(<span class="string" >'host'</span> =&gt; <span class="string" >'localhost'</span>, <span class="string" >'port'</span> =&gt; <span class="number" >12221</span>)
);
foreach($target as $t){
    $memcache = <span class="keyword" >new</span> Memcache;
    $memcache-&gt;connect($t[<span class="string" >'host'</span>], $t[<span class="string" >'port'</span>]);

    $elapsed = microtime(<span class="keyword" >true</span>);
    <span class="keyword" >for</span>($i = <span class="number" >0</span>; $i &lt; <span class="number" >1000</span>; ++$i){
        $memcache-&gt;set(<span class="string" >'hoge'</span>, <span class="string" >'123'</span>);
        $memcache-&gt;set(<span class="string" >'hoge'</span>, <span class="string" >'124'</span>);
        $memcache-&gt;get(<span class="string" >'hoge'</span>);

    }
    echo <span class="string" >'target host =&gt; '</span>, $t[<span class="string" >'host'</span>], <span class="string" >' port =&gt;'</span>, $t[<span class="string" >'port'</span>], PHP_EOL;
    echo <span class="string" >'elapsed: '</span>, (microtime(<span class="keyword" >true</span>) - $elapsed), PHP_EOL;
}</pre>

<pre class="java">target host =&gt; localhost port =&gt;<span class="number" >11211</span>
elapsed: <span class="number" >0.420136213303</span>
target host =&gt; localhost port =&gt;<span class="number" >12221</span>
elapsed: <span class="number" >1.34848499298</span>
</pre>

<p>なんつーか、「もうちょっとがんばりま賞」って感じで残念感があります。(約3倍遅い)<br />
とりあえず、動きそうなので、他の実装も頑張る。</p>
]]>
</content:encoded>
</item>

</rdf:RDF>