Pacific Side

Update:

Check out these repositories that do what is discussed in Solution 2 here:
https://github.com/trevorpowell/mapbox-gl-js-cordova-offline
https://github.com/trevorpowell/mapbox-gl-js-cordova-offline-example

Original Post:

Seen some questions about using Mapbox GL with offline vector tiles online as I've been looking into it for my own projects, and thought I might post some of what I've found. This is mainly for use cases where you might want an app for hiking, international use without data roaming, etc. There is a ticket open for offline support in the native version of Mapbox GL, but I wanted to see how it might work to use the JS version in a hybrid mobile app (Apache Cordova, AppGyver Steroids, etc.) since WebGL is supported in iOS and Android webviews now. I saw some discussion of this about two years ago with raster tiles generated in Mapbox Tilemill. My experience back then was that performance was slower when using a SQLite plugin in Cordova to load the tiles from the MBTiles database on demand, but it was better when I extracted the PNG files from the MBTiles database. Obviously, then you lose the benefits of MBTiles — deduplication of identical tiles and not having a ton of files to store. My hope was that with vector tiles being lighter weight, performance in a hybrid app might be better.

Mapbox Studio is to vector tiles what Tilemill was to raster tiles. Mapbox GL is another generation beyond Mapbox Studio — the vector tile sources are the same but the styling is different. A new style editor for Mapbox GL seems to be in the works, but for now there is a guide to follow. Mapbox Studio allows you to export vector tiles as an MBTiles database. Once exported, you can either serve the tiles from the MBTiles file or extract the tiles and serve them as PBF files. There is an example here on serving from an MBTiles file with Node.js that uses Mapbox's Tilelive. It was helpful in understanding some of how Mapbox GL and vector tiles work. Without a Node.js server in a hybrid mobile app, you would have to do things differently. Here are two possibilities on Cordova and some options for each.

Solution 1 – Extract vector tile PBF files from MBTiles database

To start with, use MBUtil to extract the vector tiles as PBF files. For example:

./mb-util --image_format=pbf test.mbtiles test

The files will be gzipped so without a way to add a gzip header when serving them, you could gunzip them:

gzip -d -r -S .pbf *

And re-add the PBF file extension:

find . -type f -exec mv '{}' '{}'.pbf \;

Appgyver Steroids runs a web server on the device that Mapbox GL JS can use when making XHR requests for the style JSON file (or you could just pass a JSON object to Mapbox GL JS for the style instead of a file reference) and the vector tile files. Plain Cordova does not have a web server, but you can use plugins like these to add one:

• https://github.com/apache/cordova-plugins/tree/master/local-webserver
• https://github.com/floatinghotpot/cordova-httpd

An alternative to using a web server plugin on Cordova might be to override the methods in Mapbox GL JS that make the calls for the style JSON and vector tiles and instead have them use the Cordova File plugin. Maybe in here:

• https://github.com/mapbox/mapbox-gl-js/blob/master/js/source/source.js
• https://github.com/mapbox/mapbox-gl-js/blob/master/js/style/style.js
• https://github.com/mapbox/mapbox-gl-js/blob/master/js/util/ajax.js

Solution 2 – Serve vector tiles from MBTiles database using SQLite plugin

This would likely be the better solution; the only question is how it would perform in a hybrid app. The idea on Cordova is to use the SQLite plugin here to retrieve the tiles from the MBTiles database:
https://github.com/brodysoft/Cordova-SQLitePlugin

Again, this would require modifying the methods in Mapbox GL JS that load the vector tiles. You may have to gunzip the tile content then and could use Pako to do so:
https://github.com/nodeca/pako

Demos

I have played around with this some using the "touch" branch from the Mapbox GL JS repository:
https://github.com/mapbox/mapbox-gl-js/tree/touch

Here are some small demos on Cordova with very basic offline vector tiles using the Natural Earth coastline data:

After the last post from this past weekend was widely shared, I was asked for further comment. This is all I would add at this point:

(1) To be fair, Wen Wei Po cited their source and the Straits Times cited their source and so on and so forth. What you have is a chain of sources of increasing credibility each quoting from a source that may be slightly less dependable. In all fairness, Wen Wei Po fairly openly, honestly, and unabashedly wrote an article around a social media report they were comfortable sharing in spite of its dubious source. The Straits Times article built on the Wen Wei Po piece by giving it a more official tone and failing to mention the social media source. From there it exploded around the world and none of the slew of articles that discussed the Wen Wei Po source mentioned the original source Wen Wei Po had cited.

(2) In my post, I was trying to draw attention to the original source of the story and the satirical nature of said source. I cannot prove beyond a shadow of a doubt regarding the veracity of the content of the source's tweet — that is, whether or not Jang was executed by dogs.

(3) We cannot prove beyond a shadow of a doubt that the Tencent account was the ultimate original source of the tweet's content or whether it too was an intermediary source, perhaps from something offline.

(4) It looks like the BBC checked with the real Choi Seongho account and he confirmed the Tencent tweet was posted by an impostor. He has since posted his own take on the execution, denying it was by dogs:
http://www.weibo.com/2834256503/AqN3YpR6U

UPDATE: Posted a response to requests for further comment on this story.

On 11 December 2013 at 02:38:07 UTC, the China-based online satirist personality known as Pyongyang Choi Seongho (or someone posing as him/her) posted a tweet to his/her Tencent Weibo account describing in detail how Kim Jongun had his uncle Jang Songtaek devoured by ravenous dogs. The original tweet can be seen here:
http://t.qq.com/p/t/312572016688539

The page features a background image of cartoon Kim Jongun giving the middle finger to his people from a balcony while flanked by senior officers. Given that the username is "choiseongho000", it's also likely this is simply a copycat account mooching off Pyongyang Choi Seongho's good name. One example of his/her style is this amusing tweet posted on U.S. Thanksgiving Day that says, "Today is Thanksgiving Day. Thanksgiving Day is the day for giving thanks to Kim Jongun. The American people eat turkey to thank him. What do you people eat?" It also includes a poster of Kim Jongun in sunglasses saying, "Don't know how to be thankful, then you'll get shot."
http://t.qq.com/p/t/310283022424717

More can be read about the Pyongyang Choi Seongho phenomenon at the links below:
http://beijingcream.com/2012/12/writer-choi-seongho-sina-weibo-star-dprk-patriot-brilliant-satirist/
http://www.wantchinatimes.com/news-subclass-cnt.aspx?id=20130418000086&cid=1103
http://www.theworldofchinese.com/article/weibos-premier-sino-satirists/
http://www.globaltimes.cn/content/775419.shtml

Even in China and North Korea it would still have been the morning of December 11 when the tweet was posted. The following day on December 12, the Wen Wei Po news source in Hong Kong published an article quoting the tweet nearly word-for-word:
http://news.wenweipo.com/2013/12/12/IN1312120039_2.htm

A screenshot of the original tweet was included with the article on Wen Wei Po. The article also cites Pyongyang Choi Seongho by name as the original source. In addition to describing how Jang Songtaek and five accomplices were stripped naked, thrown in a cage, and fed to 120 wild dogs, the article also describes how Kim Jongun removed his uncle from power for supporting his exiled oldest brother Kim Jongnam in a potential power grab. It also says that no one else had the guts to arrest his uncle so they had to have Kim's second oldest brother Kim Jongchul come out of the woodwork to personally arrest him before fading back into obscurity because he'd be perceived as a threat himself if he actually tried to hold a senior government post.

The Wen Wei Po article must have sounded plausible enough for the Straits Times in Singapore to publish the first piece in English on it on December 24:
http://www.straitstimes.com/the-big-story/asia-report/china/story/jangs-execution-bodes-ill-china-20131224

From there, the story snowballed across the mainstream English news media and it still seems to have momentum. Major English news outlets from the U.S. to the UK to India to Russia have been publishing the report:

Several have also taken a more cautious approach to the story, citing analysts and experts, but still all missing the obvious fact that the original source of the Wen Wei Po story was a tweet from a known satirist or someone posing as him/her:
http://www.nknews.org/2014/01/rumor-jang-song-thaek-was-killed-by-hungry-dogs-re-emerges/
http://www.washingtonpost.com/blogs/worldviews/wp/2014/01/03/no-kim-jong-un-probably-didnt-feed-his-uncle-to-120-hungry-dogs/
http://www.businessinsider.com/jang-song-taek-really-eaten-by-dogs-2014-1
http://www.bbc.co.uk/news/blogs-echochambers-25597324
http://blogs.telegraph.co.uk/news/timstanley/100252705/did-north-koreas-kim-jong-un-kill-his-uncle-with-120-starving-dogs/

It's amusing that given our faith in modern global news media to get to the bottom of a story, no one has actually gone back to the Wen Wei Po article and caught this. All analysis in the swaths of content that have been devoted to this report since it came out stops abruptly at a linguistic wall between the English language Straits Times story and the Chinese language Wen Wei Po article.

What do I take away from this? One, I'm reminded that language is always a barrier. Nowadays I think we imagine that global news organizations probably have multilingual experts from a wide variety of backgrounds covering all the bases. Maybe that's not the case. The ability of any one party to navigate fluidly across linguistic barriers will always be an advantage. Two, many Chinese news providers do sometimes play a little bit fast and loose with their sources when there is something that backs a viewpoint they support. Regardless of whether the tweet's content was true or false or whether the writer was aware of the source's reputation as a known satirist, Wen Wei Po saw it as something worth legitimizing. If Wen Wei Po is a government mouthpiece as some of the articles have said, then maybe that is telling. Or maybe not.

News Video Map

The most natural way of presenting world news content to me is on a map. Even without one presented to me, a geographical schema is my mind's automatic way of visualizing a list of world news articles or videos. My goal was to take a collection of world news video from providers like the AP, Reuters, and AFP, associate the content with geographic locations, and plot the results on a map.

This started as a one-day project I worked on for one of the Yahoo! Hack Days last year, and I've been playing with it since then when I have time. Hover over video clip thumbnails for a full second to open a preview of the associated video clip. Click on the play button in the preview to view the video.

The mashup combines data services and tools from the following Yahoo! providers:

It also relies on for Flash embedding and this free image resizing proxy called Images.weserv.nl.

On the backend, there's a cron job that executes the PHP layer every hour. The PHP layer takes the Yahoo! News video feed for the World News category from the past seven days and runs all the video titles and descriptions through Yahoo! Placemaker to assign geographic locations to each one in latitudinal and longitudinal coordinates. The PHP layer then creates a JavaScript array of JSON objects from the data. For production use, it then rolls up the JSON data with the required JavaScript libraries and initialization script. The JavaScript layer creates the map and plots each video with a degree of randomization to prevent video clip thumbnails from stacking on top of one another.

The major issue to overcome in development was the amount of time required for data processing. Initially, the mashup fired an AJAX request to the PHP API that returned fresh data per request. It was taking over ten seconds for the page to load each request though, so I added a five minute cache to the API. The unfortunate user who hit a stale cache still had to wait over ten seconds for the page to load. In the end, I decided that since the purpose of the service is to provide a snapshot of videos from the previous week, it's not particularly important that the data be fresh to the minute. I switched to a cron-driven script that updates the JavaScript every hour, making it available in native JSON at page load rather than via a delayed AJAX request for XML. I also made a number of optimizations to the frontend — removing duplicate scripts loaded by the Yahoo! Maps API, rolling up all required JavaScript into one file, deleting data from memory when it's no longer needed, etc. Before putting this into full-scale production I would want to use an enterprise-grade image resizing proxy to scale the video clip thumbnails before they are loaded. As it is, it's faster to load the larger thumbnails directly from the cache server than to pass them through the free image resizing proxy.

Update: Have switched to enterprise-grade image resizing. It's more reliable than the free proxy and makes everything a lot faster. :)