Replace emoji from a LUT

Add emoji for :basecamp: and :basecampy:

.gitignore

@@ -0,0 +1,2 @@
+.ruby-version
+db/NamesList.txt

.travis.yml

@@ -0,0 +1,7 @@
+script: script/test
+rvm:
+  - 1.9.3
+  - 2.1.2
+  - 2.2
+notifications:
+  email: false

CONTRIBUTING.md

@@ -0,0 +1,49 @@
+Our emoji set is based off Apple's emoji character palette, plus some custom
+emoji such as :octocat: :shipit: :metal:.
+
+Some useful tools in development are:
+
+```
+script/bootstrap
+```
+
+Sets up the development environment. The prerequisites are:
+
+* Ruby 1.9+
+* Bundler
+
+```
+rake db:generate
+```
+
+On OS X, this will rebuild the `db/Category-Emoji.json` file from the system
+one, pulling in any new emoji that Apple may have added in the meantime.
+
+```
+script/test
+```
+
+Runs the test suite, including the integrity test where we assert that we have
+covered each of Apple's emoji.
+
+```
+script/regenerate
+```
+
+Rebuilds the `db/emoji.json` file which is our main list of emoji: their
+canonical representations, descriptions, aliases, and tags. This requires OS X
+because Safari is used in the process to verify which character render as emoji
+and which render as ordinary Unicode glyphs from the current font.
+
+```
+script/console
+```
+
+Opens `irb` console with gemoji library preloded for experimentation.
+
+```
+script/release
+```
+
+For maintainers only: after the gemspec has been edited, this commits the
+change, tags a release, and pushes it to both GitHub and RubyGems.org.

Gemfile

@@ -0,0 +1,7 @@
+source "https://rubygems.org"
+
+gem "ttfunk", "~> 1.4.0"
+gem "rake"
+gem "minitest"
+
+gemspec

Gemfile.lock

@@ -0,0 +1,20 @@
+PATH
+  remote: .
+  specs:
+    gemoji (2.1.0)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    minitest (5.3.5)
+    rake (10.3.2)
+    ttfunk (1.4.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  gemoji!
+  minitest
+  rake
+  ttfunk (~> 1.4.0)

LICENSE

@@ -1,14 +1,39 @@
 octocat, squirrel, shipit
-Copyright (c) 2012 GitHub Inc. All rights reserved.
+Copyright (c) 2013 GitHub Inc. All rights reserved.
 
-bowtie, neckbeard
-Copyright (c) 2012 37signals, LLC. All rights reserved.
+bowtie, neckbeard, fu
+Copyright (c) 2013 37signals, LLC. All rights reserved.
 
 feelsgood, finnadie, goberserk, godmode, hurtrealbad, rage 1-4, suspect
-Copyright (c) 2012 id Software. All rights reserved.
+Copyright (c) 2013 id Software. All rights reserved.
 
 trollface
-Copyright (c) 2012 whynne@deviantart. All rights reserved.
+Copyright (c) 2013 whynne@deviantart. All rights reserved.
 
 All other images
-Copyright (c) 2012 Apple Inc. All rights reserved.
+Copyright (c) 2013 Apple Inc. All rights reserved.
+
+Source code:
+
+Copyright (c) 2013 GitHub, Inc.
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -5,28 +5,59 @@ Emoji images and names. See the LICENSE for copyright information.
 
 
 Installation
-============
+------------
 
-Add `gemoji` to you Gemfile.
+Add `gemoji` to your Gemfile.
 
 ``` ruby
-gem 'gemoji', :require => 'emoji/railtie'
+gem 'gemoji'
 ```
 
+**Sync images**
+
+Images can be copied to your public directory with `rake emoji` in your app. This is the recommended approach since the images will be available at a consistent location. This works best with cached formatted user content generated by tools like [html-pipeline](https://github.com/jch/html-pipeline).
+
+``` ruby
+# Rakefile
+load 'tasks/emoji.rake'
+```
+
+```
+$ rake emoji
+```
+
+**Assets Precompiling**
+
+If you must, you can manually add all the images to your asset load path.
+
+``` ruby
+# config/application.rb
+config.assets.paths << Emoji.images_path
+```
+
+Then have them compiled to public on deploy.
+
+``` ruby
+# config/application.rb
+config.assets.precompile << "emoji/**/*.png"
+```
+
+**WARNING** Since there are a ton of images, just adding the path may slow down other lookups if you aren't using it. Compiling all the emojis on deploy will add overhead to your deploy if even the images haven't changed. Theres just so many more superfluous files to iterate over. Also, the urls will be fingerprinted which may not be ideal for referencing from cached content.
+
 
 Example Rails Helper
-====================
+--------------------
 
-This would allow emojifying content such as: `it's raining :cats: and :dogs:!`
+This would allow emojifying content such as: `it's raining :cat:s and :dog:s!`
 
 See the [Emoji cheat sheet](http://www.emoji-cheat-sheet.com) for more examples.
 
 ```ruby
 module EmojiHelper
- def emojify(content)
-    h(content).to_str.gsub(/:([a-z0-9\+\-_]+):/) do |match|
-      if Emoji.names.include?($1)
-        '<img alt="' + $1 + '" height="20" src="' + asset_path("emoji/#{$1}.png") + '" style="vertical-align:middle" width="20" />'
+  def emojify(content)
+    h(content).to_str.gsub(/:([\w+-]+):/) do |match|
+      if emoji = Emoji.find_by_alias($1)
+        %(<img alt="#$1" src="#{asset_path("emoji/#{emoji.image_filename}")}" style="vertical-align:middle" width="20" height="20" />)
       else
         match
       end
@@ -34,3 +65,59 @@ module EmojiHelper
   end
 end
 ```
+
+Unicode mapping
+---------------
+
+Translate emoji names to unicode and vice versa.
+
+```ruby
+>> Emoji.find_by_alias("cat").raw
+=> "🐱"  # Don't see a cat? That's U+1F431.
+
+>> Emoji.find_by_unicode("\u{1f431}").name
+=> "cat"
+```
+
+Adding new emoji
+----------------
+
+You can add new emoji characters to the `Emoji.all` list:
+
+```ruby
+emoji = Emoji.create("music") do |char|
+  char.add_alias "song"
+  char.add_unicode_alias "\u{266b}"
+  char.add_tag "notes"
+end
+
+emoji.name #=> "music"
+emoji.raw  #=> "♫"
+emoji.image_filename #=> "unicode/266b.png"
+
+# Creating custom emoji (no Unicode aliases):
+emoji = Emoji.create("music") do |char|
+  char.add_tag "notes"
+end
+
+emoji.custom? #=> true
+emoji.image_filename #=> "music.png"
+```
+
+As you create new emoji, you must ensure that you also create and put the images
+they reference by their `image_filename` to your assets directory.
+
+For existing emojis, you can edit the list of aliases or add new tags in an edit block:
+
+```ruby
+emoji = Emoji.find_by_alias "musical_note"
+
+Emoji.edit_emoji(emoji) do |char|
+  char.add_alias "music"
+  char.add_unicode_alias "\u{266b}"
+  char.add_tag "notes"
+end
+
+Emoji.find_by_alias "music"       #=> emoji
+Emoji.find_by_unicode "\u{266b}"  #=> emoji
+```

Rakefile

@@ -0,0 +1,52 @@
+require 'rake/testtask'
+require 'rake/extensiontask'
+
+task :default => :test
+
+Rake::TestTask.new do |t|
+  t.libs << "test"
+  t.test_files = FileList["test/*_test.rb"]
+end
+
+namespace :db do
+  desc %(Generate Emoji data files needed for development)
+  task :generate => ['db/Category-Emoji.json', 'db/NamesList.txt']
+
+  desc %(Dump a list of supported Emoji with Unicode descriptions and aliases)
+  task :dump => :generate do
+    system 'ruby', '-Ilib', 'db/dump.rb'
+  end
+end
+
+emoji_plist = '/System/Library/Input Methods/CharacterPalette.app/Contents/Resources/Category-Emoji.plist'
+nameslist_url = 'http://www.unicode.org/Public/6.3.0/ucd/NamesList.txt'
+
+task 'db/Category-Emoji.json' do |t|
+  system "plutil -convert json -r '#{emoji_plist}' -o '#{t.name}'"
+end
+
+file 'db/NamesList.txt' do |t|
+  system "curl -fsSL '#{nameslist_url}' -o '#{t.name}'"
+end
+
+namespace :images do
+  desc %(Extract PNG images from Apple's "Apple Color Emoji.ttf" font)
+  task :extract do
+    require 'emoji/extractor'
+    gem_dir = File.dirname(File.realpath(__FILE__))
+    Emoji::Extractor.new(64, "#{gem_dir}/images/emoji/unicode").extract!
+  end
+end
+
+namespace :c do
+  task :headers do
+    require 'emoji/tables'
+    gem_dir = File.dirname(File.realpath(__FILE__))
+
+    File.open(File.join(gem_dir, "ext/gemoji/emoji.h"), "w") do |file|
+      file.puts(Emoji::Tables.generate_length_tables)
+    end
+  end
+end
+
+Rake::ExtensionTask.new('gemoji')

db/Category-Emoji.json

@@ -0,0 +1,47 @@
+{
+  "EmojiDataArray" : [
+    {
+      "CVDataTitle" : "EmojiCategory-People",
+      "CVCategoryImage" : "Emoji-HumanImage",
+      "CVCategoryData" : {
+        "CVSkipNullGlyphs" : true,
+        "Data" : "😄,😃,😀,😊,☺️,😉,😍,😘,😚,😗,😙,😜,😝,😛,😳,😁,😔,😌,😒,😞,😣,😢,😂,😭,😪,😥,😰,😅,😓,😩,😫,😨,😱,😠,😡,😤,😖,😆,😋,😷,😎,😴,😵,😲,😟,😦,😧,😈,👿,😮,😬,😐,😕,😯,😶,😇,😏,😑,👲,👳,👮,👷,💂,👶,👦,👧,👨,👩,👴,👵,👱,👼,👸,😺,😸,😻,😽,😼,🙀,😿,😹,😾,👹,👺,🙈,🙉,🙊,💀,👽,💩,🔥,✨,🌟,💫,💥,💢,💦,💧,💤,💨,👂,👀,👃,👅,👄,👍,👎,👌,👊,✊,✌️,👋,✋,👐,👆,👇,👉,👈,🙌,🙏,☝️,👏,💪,🚶,🏃,💃,👫,👪,👬,👭,💏,💑,👯,🙆,🙅,💁,🙋,💆,💇,💅,👰,🙎,🙍,🙇,🎩,👑,👒,👟,👞,👡,👠,👢,👕,👔,👚,👗,🎽,👖,👘,👙,💼,👜,👝,👛,👓,🎀,🌂,💄,💛,💙,💜,💚,❤️,💔,💗,💓,💕,💖,💞,💘,💌,💋,💍,💎,👤,👥,💬,👣,💭"
+      }
+    },
+    {
+      "CVDataTitle" : "EmojiCategory-Nature",
+      "CVCategoryImage" : "Emoji-NatureImage",
+      "CVCategoryData" : {
+        "CVSkipNullGlyphs" : true,
+        "Data" : "🐶,🐺,🐱,🐭,🐹,🐰,🐸,🐯,🐨,🐻,🐷,🐽,🐮,🐗,🐵,🐒,🐴,🐑,🐘,🐼,🐧,🐦,🐤,🐥,🐣,🐔,🐍,🐢,🐛,🐝,🐜,🐞,🐌,🐙,🐚,🐠,🐟,🐬,🐳,🐋,🐄,🐏,🐀,🐃,🐅,🐇,🐉,🐎,🐐,🐓,🐕,🐖,🐁,🐂,🐲,🐡,🐊,🐫,🐪,🐆,🐈,🐩,🐾,💐,🌸,🌷,🍀,🌹,🌻,🌺,🍁,🍃,🍂,🌿,🌾,🍄,🌵,🌴,🌲,🌳,🌰,🌱,🌼,🌐,🌞,🌝,🌚,🌑,🌒,🌓,🌔,🌕,🌖,🌗,🌘,🌜,🌛,🌙,🌍,🌎,🌏,🌋,🌌,🌠,⭐️,☀️,⛅️,☁️,⚡️,☔️,❄️,⛄️,🌀,🌁,🌈,🌊"
+      }
+    },
+    {
+      "CVDataTitle" : "EmojiCategory-Objects",
+      "CVCategoryImage" : "Emoji-ObjectsImage",
+      "CVCategoryData" : {
+        "CVSkipNullGlyphs" : true,
+        "Data" : "🎍,💝,🎎,🎒,🎓,🎏,🎆,🎇,🎐,🎑,🎃,👻,🎅,🎄,🎁,🎋,🎉,🎊,🎈,🎌,🔮,🎥,📷,📹,📼,💿,📀,💽,💾,💻,📱,☎️,📞,📟,📠,📡,📺,📻,🔊,🔉,🔈,🔇,🔔,🔕,📢,📣,⏳,⌛️,⏰,⌚️,🔓,🔒,🔏,🔐,🔑,🔎,💡,🔦,🔆,🔅,🔌,🔋,🔍,🛁,🛀,🚿,🚽,🔧,🔩,🔨,🚪,🚬,💣,🔫,🔪,💊,💉,💰,💴,💵,💷,💶,💳,💸,📲,📧,📥,📤,✉️,📩,📨,📯,📫,📪,📬,📭,📮,📦,📝,📄,📃,📑,📊,📈,📉,📜,📋,📅,📆,📇,📁,📂,✂️,📌,📎,✒️,✏️,📏,📐,📕,📗,📘,📙,📓,📔,📒,📚,📖,🔖,📛,🔬,🔭,📰,🎨,🎬,🎤,🎧,🎼,🎵,🎶,🎹,🎻,🎺,🎷,🎸,👾,🎮,🃏,🎴,🀄️,🎲,🎯,🏈,🏀,⚽️,⚾️,🎾,🎱,🏉,🎳,⛳️,🚵,🚴,🏁,🏇,🏆,🎿,🏂,🏊,🏄,🎣,☕️,🍵,🍶,🍼,🍺,🍻,🍸,🍹,🍷,🍴,🍕,🍔,🍟,🍗,🍖,🍝,🍛,🍤,🍱,🍣,🍥,🍙,🍘,🍚,🍜,🍲,🍢,🍡,🍳,🍞,🍩,🍮,🍦,🍨,🍧,🎂,🍰,🍪,🍫,🍬,🍭,🍯,🍎,🍏,🍊,🍋,🍒,🍇,🍉,🍓,🍑,🍈,🍌,🍐,🍍,🍠,🍆,🍅,🌽"
+      }
+    },
+    {
+      "CVDataTitle" : "EmojiCategory-Places",
+      "CVCategoryImage" : "Emoji-PlacesImage",
+      "CVCategoryData" : {
+        "CVSkipNullGlyphs" : true,
+        "Data" : "🏠,🏡,🏫,🏢,🏣,🏥,🏦,🏪,🏩,🏨,💒,⛪️,🏬,🏤,🌇,🌆,🏯,🏰,⛺️,🏭,🗼,🗾,🗻,🌄,🌅,🌃,🗽,🌉,🎠,🎡,⛲️,🎢,🚢,⛵️,🚤,🚣,⚓️,🚀,✈️,💺,🚁,🚂,🚊,🚉,🚞,🚆,🚄,🚅,🚈,🚇,🚝,🚋,🚃,🚎,🚌,🚍,🚙,🚘,🚗,🚕,🚖,🚛,🚚,🚨,🚓,🚔,🚒,🚑,🚐,🚲,🚡,🚟,🚠,🚜,💈,🚏,🎫,🚦,🚥,⚠️,🚧,🔰,⛽️,🏮,🎰,♨️,🗿,🎪,🎭,📍,🚩,🇯🇵,🇰🇷,🇩🇪,🇨🇳,🇺🇸,🇫🇷,🇪🇸,🇮🇹,🇷🇺,🇬🇧"
+      }
+    },
+    {
+      "CVDataTitle" : "EmojiCategory-Symbols",
+      "CVCategoryImage" : "Emoji-SymbolImage",
+      "CVCategoryData" : {
+        "CVSkipNullGlyphs" : true,
+        "Data" : "1️⃣,2️⃣,3️⃣,4️⃣,5️⃣,6️⃣,7️⃣,8️⃣,9️⃣,0️⃣,🔟,🔢,#️⃣,🔣,⬆️,⬇️,⬅️,➡️,🔠,🔡,🔤,↗️,↖️,↘️,↙️,↔️,↕️,🔄,◀️,▶️,🔼,🔽,↩️,↪️,ℹ️,⏪,⏩,⏫,⏬,⤵️,⤴️,🆗,🔀,🔁,🔂,🆕,🆙,🆒,🆓,🆖,📶,🎦,🈁,🈯️,🈳,🈵,🈴,🈲,🉐,🈹,🈺,🈶,🈚️,🚻,🚹,🚺,🚼,🚾,🚰,🚮,🅿️,♿️,🚭,🈷,🈸,🈂,Ⓜ️,🛂,🛄,🛅,🛃,🉑,㊙️,㊗️,🆑,🆘,🆔,🚫,🔞,📵,🚯,🚱,🚳,🚷,🚸,⛔️,✳️,❇️,❎,✅,✴️,💟,🆚,📳,📴,🅰,🅱,🆎,🅾,💠,➿,♻️,♈️,♉️,♊️,♋️,♌️,♍️,♎️,♏️,♐️,♑️,♒️,♓️,⛎,🔯,🏧,💹,💲,💱,©,®,™,❌,‼️,⁉️,❗️,❓,❕,❔,⭕️,🔝,🔚,🔙,🔛,🔜,🔃,🕛,🕧,🕐,🕜,🕑,🕝,🕒,🕞,🕓,🕟,🕔,🕠,🕕,🕖,🕗,🕘,🕙,🕚,🕡,🕢,🕣,🕤,🕥,🕦,✖️,➕,➖,➗,♠️,♥️,♣️,♦️,💮,💯,✔️,☑️,🔘,🔗,➰,〰,〽️,🔱,◼️,◻️,◾️,◽️,▪️,▫️,🔺,🔲,🔳,⚫️,⚪️,🔴,🔵,🔻,⬜️,⬛️,🔶,🔷,🔸,🔹"
+      }
+    }
+  ],
+  "CVViewFontList" : [
+    "AppleColorEmoji"
+  ]
+}

db/aliases.applescript

@@ -0,0 +1,13 @@
+set jsCode to "document.getElementById('output').value"
+set json to missing value
+
+tell application "Safari"
+  repeat
+    set json to (do JavaScript jsCode in current tab of window 1)
+    if (json is not missing value) then exit repeat
+    delay 0.5
+  end repeat
+  close current tab of window 1
+end tell
+
+return json

db/aliases.html

@@ -0,0 +1,81 @@
+<!DOCTYPE html>
+<title>Emoji alias detection</title>
+<style>
+  textarea {
+    font-family: monospace;
+  }
+</style>
+
+<p>Save the following as <tt>emoji.json</tt>:</p>
+
+<textarea id="output" rows="50" cols="80"></textarea>
+
+<script>
+  const VARIATION_SELECTOR_15 = String.fromCharCode(0xfe0e);
+  const VARIATION_SELECTOR_16 = String.fromCharCode(0xfe0f);
+  const EMOJI_SIZE = 32
+
+  function detectAliases(db) {
+    for (var i = 0; i < db.length; ++i) {
+      var emoji = db[i];
+      var raw = emoji.emoji;
+      if (!raw) {
+        continue;
+      }
+
+      if (raw.indexOf(VARIATION_SELECTOR_16) > -1) {
+        var candidates = [raw.replace(VARIATION_SELECTOR_16, ""), raw];
+      } else {
+        var candidates = [raw, raw + VARIATION_SELECTOR_16];
+      }
+
+      var aliases = candidates.filter(isColorEmoji);
+      emoji.emoji = aliases[0];
+    }
+
+    dump(db);
+  }
+
+  function isColorEmoji(candidate) {
+    // Draw the emoji twice using a different color each time. If the emoji
+    // draws as the same color regardless of what color we set, it's a color
+    // emoji.
+    return color(candidate, "#f00") === color(candidate, "#0f0");
+  }
+
+  var canvas = document.createElement("canvas");
+  canvas.width = canvas.height = EMOJI_SIZE;
+
+  function color(emoji, rgb) {
+    var context = canvas.getContext("2d");
+    context.clearRect(0, 0, canvas.width, canvas.height);
+    context.fillStyle = rgb;
+    context.textBaseline = "top";
+    context.font = EMOJI_SIZE+"px Arial";
+    context.fillText(emoji, 0, 0);
+    var data = context.getImageData(0, 0, EMOJI_SIZE, EMOJI_SIZE).data;
+    for (var i = 0; i < data.length; i += 4) {
+      if (data[i] === 0 && data[i + 1] === 0 && data[i + 2] === 0) {
+        continue;
+      }
+      return data[i].toString(16)
+        + data[i + 1].toString(16)
+        + data[i + 2].toString(16);
+    }
+    return "no colored pixel found";
+  }
+
+  function dump(db) {
+    var json = JSON.stringify(db, null, "  ")
+      .replace(/^( +)(.+)\[\](,?)$/mg, "$1$2[\n$1]$3")
+      .replace(/,\n( *)  /g, "\n$1, ");
+    document.getElementById("output").value = json + "\n";
+  }
+
+  var xhr = new XMLHttpRequest;
+  xhr.onload = function() {
+    detectAliases(JSON.parse(this.responseText));
+  };
+  xhr.open("GET", "emoji.json", false);
+  xhr.send(null);
+</script>

db/dump.rb

@@ -0,0 +1,74 @@
+require 'emoji'
+require 'json'
+
+names_list = File.expand_path('../NamesList.txt', __FILE__)
+
+class UnicodeCharacter
+  attr_reader :code, :description, :aliases
+
+  @index = {}
+  class << self
+    attr_reader :index
+
+    def fetch(code, *args, &block)
+      code = code.to_s(16).rjust(4, '0') if code.is_a?(Integer)
+      index.fetch(code, *args, &block)
+    end
+  end
+
+  def initialize(code, description)
+    @code = code.downcase
+    @description = description.downcase
+    @aliases = []
+    @references = []
+
+    self.class.index[@code] = self
+  end
+
+  def add_alias(string)
+    @aliases.concat string.split(/\s*,\s*/)
+  end
+
+  def add_reference(code)
+    @references << code.downcase
+  end
+end
+
+char = nil
+
+File.foreach(names_list) do |line|
+  case line
+  when /^[A-F0-9]{4,5}\t/
+    code, desc = line.chomp.split("\t", 2)
+    codepoint = code.hex
+    char = UnicodeCharacter.new(code, desc)
+  when /^\t= /
+    char.add_alias($')
+  when /^\tx .+ - ([A-F0-9]{4,5})\)$/
+    char.add_reference($1)
+  end
+end
+
+trap(:PIPE) { abort }
+
+items = []
+variation_codepoint = Emoji::VARIATION_SELECTOR_16.codepoints[0]
+
+for emoji in Emoji.all
+  item = {}
+
+  unless emoji.custom?
+    chars = emoji.raw.codepoints.map { |code| UnicodeCharacter.fetch(code) unless code == variation_codepoint }.compact
+    item[:emoji] = emoji.raw
+    item[:description] = chars.map(&:description).join(' + ')
+  end
+
+  item[:aliases] = emoji.aliases
+  item[:tags] = emoji.tags
+
+  items << item
+end
+
+puts JSON.pretty_generate(items)
+  .gsub("\n\n", "\n")
+  .gsub(/,\n( +)/) { "\n%s, " % $1[2..-1] }

db/index.html

@@ -0,0 +1,113 @@
+<!doctype html>
+<title>Emoji preview</title>
+
+<style type="text/css">
+body {
+  font: 14px/18px Verdana, Arial, sans-serif;
+  padding: 2em;
+}
+ol {
+  list-style: none;
+  padding-left: 0;
+}
+li {
+  margin-left: 0;
+  margin-top: 5px;
+  clear: left;
+}
+li:first-child { display: none; }
+li > span {
+  display: block;
+}
+.emoji {
+  font-size: 50px;
+  line-height: 50px;
+  float: left;
+  margin-right: 20px;
+}
+.description {
+  font-style: italic;
+  color: gray;
+}
+.aliases span:before, .aliases span:after {
+  content: ":";
+  color: gray;
+}
+.tags {
+  font-size: 11px;
+}
+.tags span {
+  display: inline-block;
+  padding: 1px 5px 2px;
+  border-radius: 3px;
+  background: #dadada;
+  line-height: 11px;
+}
+</style>
+
+<p><input type=search placeholder="Search..."></p>
+
+<ol>
+  <li>
+    <span class="emoji"></span>
+    <span class="description"></span>
+    <span class="aliases"></span>
+    <span class="tags"></span>
+  </li>
+</ol>
+
+<script>
+  function renderEmoji(emojis) {
+    var item, els, template = document.querySelector('li')
+    for (var emoji, i=0; i < emojis.length; i++) {
+      emoji = emojis[i]
+      item = template.cloneNode(true)
+      els = item.querySelectorAll('span')
+      if (emoji.emoji) els[0].textContent = emoji.emoji
+      else {
+        var img = document.createElement('img')
+        img.src = "../images/emoji/" + emoji.aliases[0] + ".png"
+        els[0].appendChild(img)
+      }
+      els[1].textContent = emoji.description || ''
+      els[2].innerHTML = emoji.aliases.map(function(n){ return '<span>'+n+'</span>' }).join(' ')
+      els[3].innerHTML = emoji.tags.map(function(n){ return '<span>'+n+'</span>' }).join(' ')
+      template.parentNode.appendChild(item)
+
+      item._emojiText = (els[2].textContent + ' ' + els[3].textContent).replace(/_/g, '-')
+    }
+  }
+
+  xhr = new XMLHttpRequest
+  xhr.onreadystatechange = function() {
+    if (this.readyState == 4) {
+      json = JSON.parse(this.responseText)
+      renderEmoji(json)
+    }
+  }
+  xhr.open('GET', 'emoji.json', false)
+  xhr.send(null)
+
+  function searchEmoji(query) {
+    var el,
+        re = new RegExp('\\b' + query),
+        els = document.querySelectorAll('li')
+
+    for (var i=1; i < els.length; i++) {
+      el = els[i]
+      if ( !query || re.test(el._emojiText) ) {
+        el.style.display = 'list-item'
+      } else {
+        el.style.display = 'none'
+      }
+    }
+  }
+
+  var timeout, search = document.querySelector('input[type=search]')
+  search.addEventListener('input', function() {
+    var $this = this
+    if (timeout) clearTimeout(timeout)
+    timeout = setTimeout(function(){ searchEmoji($this.value) }, 200)
+  }, false)
+  search.focus()
+</script>

ext/gemoji/emoji.h

@@ -0,0 +1,11 @@
+static const long emoji_byte_lengths[][5] = {
+	{0},
+	{11, 8, 7, 4, 0},
+	{6, 3, 0},
+	{7, 0},
+	{6, 0},
+	{5, 0},
+};
+
+static const int8_t emoji_magic_bytes[] =
+{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

ext/gemoji/extconf.rb

@@ -0,0 +1,6 @@
+require 'mkmf'
+
+$CFLAGS << ' -ggdb3 -O0 '
+
+dir_config('gemoji')
+create_makefile('gemoji')

ext/gemoji/gemoji.c

@@ -0,0 +1,125 @@
+#include <ruby.h>
+#include <ruby/encoding.h>
+#include "emoji.h"
+
+#define unlikely(x) __builtin_expect((x),0)
+
+static long
+lookup_emoji(VALUE *rb_emoji, VALUE rb_bytes, VALUE rb_unicode_map, const long *possible_len)
+{
+	const uint8_t *src = (uint8_t *)RSTRING_PTR(rb_bytes);
+
+	for (; *possible_len; ++possible_len) {
+		const long emoji_size = *possible_len;
+
+		if (emoji_size > RSTRING_LEN(rb_bytes))
+			continue;
+
+		if ((src[emoji_size - 1] & 0xC0) != 0x80)
+			continue;
+
+		rb_str_set_len(rb_bytes, emoji_size);
+
+		*rb_emoji = rb_hash_lookup(rb_unicode_map, rb_bytes);
+		if (!NIL_P(*rb_emoji))
+			return emoji_size;
+	}
+
+	return 0;
+}
+
+static VALUE
+replace_emoji(const uint8_t *src, long size, VALUE rb_unicode_map)
+{
+	VALUE rb_emoji, rb_bytes, rb_out = Qnil;
+	long i = 0, org, emoji_len;
+	int8_t emoji_byte;
+
+	while (i < size) {
+		org = i;
+
+retry_search:
+		while (i < size && (emoji_byte = emoji_magic_bytes[(int)src[i]]) == 0)
+			i++;
+
+		if (i + 1 < size && (src[i + 1] & 0x80) != 0x80) {
+			i++;
+			goto retry_search;
+		}
+
+		if (unlikely(org == 0)) {
+			if (i == size)
+				return Qnil;
+
+			rb_out = rb_str_buf_new(size * 4 / 3);
+			rb_enc_associate(rb_out, rb_utf8_encoding());
+
+			rb_bytes = rb_str_buf_new(16);
+			rb_enc_associate(rb_bytes, rb_utf8_encoding());
+		}
+
+		if (i > org)
+			rb_str_buf_cat(rb_out, (const char *)src + org, i - org);
+
+		if (unlikely(i == size))
+			break;
+
+		emoji_len = size - i;
+		if (emoji_len > 12)
+			emoji_len = 12;
+
+		memcpy(RSTRING_PTR(rb_bytes), src + i, emoji_len);
+		rb_str_set_len(rb_bytes, emoji_len);
+
+		emoji_len = lookup_emoji(&rb_emoji, rb_bytes, rb_unicode_map, emoji_byte_lengths[(int)emoji_byte]);
+
+		if (emoji_len) {
+			VALUE rb_repl = rb_yield(rb_emoji);
+
+			if (NIL_P(rb_repl)) {
+				rb_str_buf_cat(rb_out, (const char *)src + i, emoji_len);
+			} else {
+				Check_Type(rb_repl, T_STRING);
+				rb_str_buf_append(rb_out, rb_repl);
+			}
+
+			i += emoji_len;
+			continue;
+		}
+
+		rb_str_buf_cat(rb_out, (const char *)src + i, 1);
+		i++;
+	}
+
+	return rb_out;
+}
+
+static VALUE
+rb_gemoji_replace_unicode(VALUE klass, VALUE rb_source)
+{
+	VALUE rb_output;
+	VALUE rb_unicode_map = rb_funcall(klass, rb_intern("unicodes_index"), 0);
+
+	Check_Type(rb_source, T_STRING);
+	Check_Type(rb_unicode_map, T_HASH);
+
+	rb_must_asciicompat(rb_source);
+
+	if (ENC_CODERANGE_ASCIIONLY(rb_source))
+		return rb_source;
+
+	if (rb_enc_get(rb_source) != rb_utf8_encoding())
+		rb_raise(rb_eEncCompatError, "expected UTF-8 encoding");
+
+	rb_output = replace_emoji((uint8_t *)RSTRING_PTR(rb_source), RSTRING_LEN(rb_source), rb_unicode_map);
+	if (NIL_P(rb_output))
+		return rb_source;
+
+	return rb_output;
+}
+
+void Init_gemoji(void)
+{
+	VALUE rb_mEmoji = rb_define_module("Emoji");
+	rb_define_method(rb_mEmoji, "gsub_unicode", rb_gemoji_replace_unicode, 1);
+}

gemoji.gemspec

@@ -1,12 +1,21 @@
 Gem::Specification.new do |s|
   s.name    = "gemoji"
-  s.version = "1.1.0"
-  s.summary = "Emoji Assets"
-  s.description = "Emoji assets"
+  s.version = "2.1.0"
+  s.summary = "Emoji conversion and image assets"
+  s.description = "Image assets and character information for emoji."
+
+  s.required_ruby_version = '> 1.9'
 
   s.authors  = ["GitHub"]
   s.email    = "support@github.com"
   s.homepage = "https://github.com/github/gemoji"
+  s.licenses = ["MIT"]
 
-  s.files = Dir["README.md", "images/**/*.png", "lib/**/*.rb", "lib/tasks/*.rake"]
+  s.files = Dir[
+    "README.md",
+    "images/**/*.png",
+    "db/emoji.json",
+    "lib/**/*.rb",
+    "lib/tasks/*.rake"
+  ]
 end

images/emoji/+1.png

@@ -1 +0,0 @@
-unicode/e00e.png

images/emoji/-1.png

@@ -1 +0,0 @@
-unicode/e421.png

images/emoji/100.png

@@ -1 +0,0 @@
-unicode/1f4af.png

images/emoji/109.png

@@ -1 +0,0 @@
-unicode/e50a.png

images/emoji/1234.png

@@ -1 +0,0 @@
-unicode/1f522.png

images/emoji/8ball.png

@@ -1 +0,0 @@
-unicode/e42c.png

images/emoji/a.png

@@ -1 +0,0 @@
-unicode/e532.png

images/emoji/ab.png

@@ -1 +0,0 @@
-unicode/e534.png

images/emoji/abc.png

@@ -1 +0,0 @@
-unicode/1f524.png

images/emoji/abcd.png

@@ -1 +0,0 @@
-unicode/1f521.png

images/emoji/accept.png

@@ -1 +0,0 @@
-unicode/1f251.png

images/emoji/aerial_tramway.png

@@ -1 +0,0 @@
-unicode/1f6a1.png

images/emoji/airplane.png

@@ -1 +0,0 @@
-unicode/e01d.png

images/emoji/alarm_clock.png

@@ -1 +0,0 @@
-unicode/23f0.png

images/emoji/alien.png

@@ -1 +0,0 @@
-unicode/e10c.png

images/emoji/ambulance.png

@@ -1 +0,0 @@
-unicode/e431.png

images/emoji/anchor.png

@@ -1 +0,0 @@
-unicode/2693.png

images/emoji/angel.png

@@ -1 +0,0 @@
-unicode/e04e.png

images/emoji/anger.png

@@ -1 +0,0 @@
-unicode/e334.png

images/emoji/angry.png

@@ -1 +0,0 @@
-unicode/e059.png

images/emoji/ant.png

@@ -1 +0,0 @@
-unicode/1f41c.png

images/emoji/apple.png

@@ -1 +0,0 @@
-unicode/e345.png

images/emoji/aquarius.png

@@ -1 +0,0 @@
-unicode/e249.png

images/emoji/aries.png

@@ -1 +0,0 @@
-unicode/e23f.png

images/emoji/arrow_backward.png

@@ -1 +0,0 @@
-unicode/e23b.png

images/emoji/arrow_double_down.png

@@ -1 +0,0 @@
-unicode/23ec.png

images/emoji/arrow_double_up.png

@@ -1 +0,0 @@
-unicode/23eb.png

images/emoji/arrow_down.png

@@ -1 +0,0 @@
-unicode/e233.png

images/emoji/arrow_down_small.png

@@ -1 +0,0 @@
-unicode/1f53d.png

images/emoji/arrow_forward.png

@@ -1 +0,0 @@
-unicode/e23a.png

images/emoji/arrow_heading_down.png

@@ -1 +0,0 @@
-unicode/2935.png

images/emoji/arrow_heading_up.png

@@ -1 +0,0 @@
-unicode/2934.png

images/emoji/arrow_left.png

@@ -1 +0,0 @@
-unicode/e235.png

images/emoji/arrow_lower_left.png

@@ -1 +0,0 @@
-unicode/e239.png

images/emoji/arrow_lower_right.png

@@ -1 +0,0 @@
-unicode/e238.png

images/emoji/arrow_right.png

@@ -1 +0,0 @@
-unicode/e234.png

images/emoji/arrow_right_hook.png

@@ -1 +0,0 @@
-unicode/21aa.png

images/emoji/arrow_up.png

@@ -1 +0,0 @@
-unicode/e232.png

images/emoji/arrow_up_down.png

@@ -1 +0,0 @@
-unicode/2195.png

images/emoji/arrow_up_small.png

@@ -1 +0,0 @@
-unicode/1f53c.png

images/emoji/arrow_upper_left.png

@@ -1 +0,0 @@
-unicode/e237.png

images/emoji/arrow_upper_right.png

@@ -1 +0,0 @@
-unicode/e236.png

images/emoji/arrows_clockwise.png

@@ -1 +0,0 @@
-unicode/1f503.png

images/emoji/arrows_counterclockwise.png

@@ -1 +0,0 @@
-unicode/1f504.png

images/emoji/art.png

@@ -1 +0,0 @@
-unicode/e502.png

images/emoji/articulated_lorry.png

@@ -1 +0,0 @@
-unicode/1f69b.png

images/emoji/astonished.png

@@ -1 +0,0 @@
-unicode/e410.png

images/emoji/atm.png

@@ -1 +0,0 @@
-unicode/e154.png

images/emoji/b.png

@@ -1 +0,0 @@
-unicode/e533.png

images/emoji/baby.png

@@ -1 +0,0 @@
-unicode/e51a.png

images/emoji/baby_bottle.png

@@ -1 +0,0 @@
-unicode/1f37c.png

images/emoji/baby_chick.png

@@ -1 +0,0 @@
-unicode/e523.png

images/emoji/baby_symbol.png

@@ -1 +0,0 @@
-unicode/e13a.png

images/emoji/baggage_claim.png

@@ -1 +0,0 @@
-unicode/1f6c4.png

images/emoji/balloon.png

@@ -1 +0,0 @@
-unicode/e310.png

images/emoji/ballot_box_with_check.png

@@ -1 +0,0 @@
-unicode/2611.png

images/emoji/bamboo.png

@@ -1 +0,0 @@
-unicode/e436.png

images/emoji/banana.png

@@ -1 +0,0 @@
-unicode/1f34c.png

images/emoji/bangbang.png

@@ -1 +0,0 @@
-unicode/203c.png

images/emoji/bank.png

@@ -1 +0,0 @@
-unicode/e14d.png

images/emoji/bar_chart.png

@@ -1 +0,0 @@
-unicode/1f4ca.png

images/emoji/barber.png

@@ -1 +0,0 @@
-unicode/e320.png

images/emoji/baseball.png

@@ -1 +0,0 @@
-unicode/e016.png

images/emoji/basketball.png

@@ -1 +0,0 @@
-unicode/e42a.png

images/emoji/bath.png

@@ -1 +0,0 @@
-unicode/e13f.png

images/emoji/bathtub.png

@@ -1 +0,0 @@
-unicode/1f6c1.png

images/emoji/battery.png

@@ -1 +0,0 @@
-unicode/1f50b.png

images/emoji/bear.png

@@ -1 +0,0 @@
-unicode/e051.png

images/emoji/beer.png

@@ -1 +0,0 @@
-unicode/e047.png

images/emoji/beers.png

@@ -1 +0,0 @@
-unicode/e30c.png

images/emoji/beetle.png

@@ -1 +0,0 @@
-unicode/1f41e.png

images/emoji/beginner.png

@@ -1 +0,0 @@
-unicode/e209.png

images/emoji/bell.png

@@ -1 +0,0 @@
-unicode/e325.png

images/emoji/bento.png

@@ -1 +0,0 @@
-unicode/e34c.png

images/emoji/bicyclist.png

@@ -1 +0,0 @@
-unicode/1f6b4.png

images/emoji/bike.png

@@ -1 +0,0 @@
-unicode/e136.png

images/emoji/bikini.png

@@ -1 +0,0 @@
-unicode/e322.png

images/emoji/bird.png

@@ -1 +0,0 @@
-unicode/e521.png

images/emoji/birthday.png

@@ -1 +0,0 @@
-unicode/e34b.png

images/emoji/black_circle.png

@@ -1 +0,0 @@
-unicode/26ab.png